See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/4307102

Voice controlled intelligent wheelchair

Conference Paper · October 2007

DOI: 10.1109/SICE.2007.4421003 · Source: IEEE Xplore

CITATIONS READS

39 9,518

3 authors, including:

Takeshi Saitoh
Kyushu Institute of Technology
48 PUBLICATIONS   502 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Visual Speech Recognition (VSR) View project

Gaze Estimation View project

All content following this page was uploaded by Takeshi Saitoh on 21 May 2014.

The user has requested enhancement of the downloaded file.

SICE Annual Conference 2007
Sept. 17-20, 2007, Kagawa University, Japan

Voice Controlled Intelligent Wheelchair

Masato Nishimori, Takeshi Saitoh and Ryosuke Konishi
Tottori University, Tottori, Japan
E-mail: {saitoh, konishi}@ele.tottori-u.ac.jp

Abstract: In order to assist physically handicapped persons, we developed a voice controlled wheelchair. The user can
control the wheelchair by voice commands, such as “susume (run forward)” in Japanese. A grammar-based recognition
parser named “Julian” is used in our system. Three type commands, the basic reaction command, the short moving
reaction command, and the verification command, are given. We experimented speech recognition by Julian, and obtained
a successful recognition rate 98.3%, 97.0% of the movement command and the verification command, respectively. The
running experiment with three persons was carried out in the campus room and the utility of our system is shown.

Keywords: voice control, intelligent wheelchair, speech recognition.

1. INTRODUCTION 2. VOICE CONTROLLED INTELLIGENT

Recently, the old person and the physically hand-
icapped person who use a wheelchair are increasing.
However, only two type wheelchairs by the hand-
operating and operating the joystick, have come into wide
use. The former type needs muscular strength for the op-
eration and the latter type needs the skill. Therefore, there
is a problem that it is difficult for the old and the handi-
capped person to use these interfaces.
The interface of wheelchair, voice [1], [2], direction of
the face [3], eye gaze [4], electromyography (EMG) sig-
nal from the neck and the arm muscles [5], EMG signal
from the face muscles [6], wireless tongue-palate con-
tact pressure sensor [7], eye-control method based on
electrooculography (EOG) [8], electroencephalography
(EEG) [9] are proposed. Using these methods, it be-
comes easy to physically a handicapped person, to op-
erate a wheelchair.
Though a person moves his face and the direction of
his eyes unconsciously, unnaturalness causes to use these
movements as the interface consciously. Operation by
EMG or EOG or EEG signal needs the skill, and a mental
burden is forced on the user to use a contact type sensor
to measure these signals. On the other hand, voice is a
natural communication way, and voice is one of the easy
interfaces.
This research develops a voice controlled wheelchair
which uses the voice command as the interface. The
basic modes of the wheelchair is, 1) running until next
command is input, 2) turning or rotating, and 3) stop-
ping. Though, our system has these modes, in this pa-
per, to use in the narrow space, we use four commands
which move short distance. Moreover, we also use two
type verification (acceptance and rejection) commands to
prevent the wrong reaction by the misrecognition. The
running experiments are carried out with some persons in
the campus corridor or room, and the utility of our system
is shown.
WHEELCHAIR
2.1 Voice command and wheelchair reaction
We set voice commands to control wheelchair. The
voice commands consist of nine reaction commands and
five verification commands. The reaction commands con-
sist of five basic reaction commands and four short mov-
ing reaction commands which move short distance. The
details of voice commands and their respective reaction
are shown in table 1. In this table,from ⃝ 1 to ⃝5 are the
basic reaction commands, from ⃝ 6 to ⃝9 are the short
moving reaction commands, and ⃝ A and ⃝ B are the ver-
ification commands.
Two commands “migi” and “hidari” are the 90 de-
grees rotational movement in a fixed place without any
forward or backward movement. When these commands
are input during running, the system turns 90 degrees
and move forward after turn. Thus, five basic reaction
commands achieve seven reactions (modes). In our sys-
tem, though the short moving reactions, such as forward
running about 30cm and right rotating about 30 degree,
are decided in experience, these reactions is possible to
change by user’s needs.
After voice command is input, recognition process is
carried out, and recognition result is shown in the display
of the laptop. In this time, to avoid wrong reaction by the
misrecognition, the user must input a verification com-
mand to check the result. When acceptance word (“OK”
or “yes”) is input, the system considers the recognition
result as correct, and it reacts to the given movement.
Oppositely, when rejection word (“torikeshi” or “no” or
“cancel”) is input, or no voice for a while, the system
considers the recognition result as incorrect.
2.2 System configuration
Our system is based on commercial electronic
wheelchair Nissin Medical Industries co. NEO-P1. Our
system consists of a headset microphone and a laptop.
The overview and the construction of our system are
shown in figure 1 and figure 2, respectively. In our sys-
tem, the user can select two type controls voice and but-
ton. Since the main issue of this paper is voice controlled

- 336 -
PR0001/07/0000-0336 ¥400 © 2007 SICE
Table 1 Voice command and reaction.
START
command reaction (mode)
⃝
1 tomare stop input reaction command
⃝
2 susume run forward
⃝
3 sagare run backward recognition
⃝
4 migi turn right / rotate right (reaction command)
⃝
5 hidari turn left / rotate left
⃝
6 sukoshi-susume run forward about 30cm show result word
⃝
7 sukoshi-sagare run backward about 30cm Yes
⃝
8 sukoshi-migi rotate right about 30 degrees stop command
⃝
9 sukoshi-hidari rotate left about 30 degrees No
Yes
⃝
A OK / yes acceptance command no input for 3 seconds
⃝
B torikeshi / no / cancel rejection command No
input verification command

recognition
(verification command)

No
OK?
Yes

reaction

Fig. 3 Control algorithm.

2.3 Control algorithm

Fig. 1 System overview.
system, this paper does not refer to button control. The
laptop is main computer of this system, and recognition
process is executed here. The control signal is sent to
PIC from the laptop, and PIC generates the motor control
signal to drive the wheelchair.
We use a grammar-based recognition parser named
“Julian”. This is open-source software and is developed
by Kyoto University, Nara Institute of Science and Tech-
nology, and so on [10]. Julian is another version of Julius
which performs grammar-based recognition. Julius is an
open-source large vocabulary continuous speech recogni-
tion engine. Julian can perform speech recognition based
on a written grammar. The grammar describes possible
syntax or patterns of words on a specific task. When a
speech input is given, Julian searches for the most likely
word sequence under constraint of the given grammar.
Furthermore, in our system, the forward running speed
is set at 1.8 km/h, and backward speed is set at 1.4 km/h.
laptop
microphone control system
voice
command audio recognition
signal (reaction command)
button control recognition
(verification command)
Fig. 2 Voice controlled intelligent wheelchair.
To control the wheelchair by voice command, a user
inputs reaction command and verification command to
prevent wrong reaction by the misrecognition. In our sys-
tem, the control algorithm is shown in figure 3 and detail
is as follows:
1. User inputs a reaction command.
2. Input reaction voice data is recognized and its result
word is displayed.
3. When result word is a stop command, go to 7.
4. User inputs a verification command. When any ver-
ification command is not input within three seconds,
the system considers that recognition result fails and
go to 1.
5. Input verification voice data is recognized.
6. When result word is a rejection command, go to 1.
7. Control system sends the target control signal to the
wheelchair, and the wheelchair reacts. Go to 1.
2.4 Main Window
A user operates control system with riding a
wheelchair. The window of this system which is simple
and easy to understand is desirable. Then, the window
is designed that many illustrations are used and little text
are used. Our created window is shown in figure 4.
Each illustration and button inside the main window
wheelchair
has the following function.
PIC
1. Audio wave: input audio signal is displayed in real-
left motor
time.
right motor 2. Mode illustration: recognition result is displayed as
control
signal an illustration of corresponding mode. Nine mode
illustration is shown in figure 5.
3. Start button: When a button is pushed, the input of
the audio signal is started.

- 337 -
 ex.1 A user inputs stop command to stop in the target
position by going straight running.
2 ex.2 A user inputs stop command in the target position
1 by going straight running.
ex.3 A user makes a wheelchair go straight at the time
5 of the stop, immediately he inputs stop command.
The stopping experiment was carried out with the five
3 persons. Each person was tested five times. Thus, aver-
age value and standard deviation were calculated. Exper-
imental results are shown in figure 7.
4
As for ex.1, average error was 11.7cm. However, we
found that increasing number of time of the experiment
decreases the error.
From two experimental results of ex.2 and ex.3, the
Fig. 4 Main window.
braking distance of about 2m occurred until it actually
stopped after stop command was input. This is the reason
why two seconds are necessary for the recognition pro-
cessing and the display of the result. The running speed is
1.8km/h, and our system runs 2m in 2 seconds. This dis-
⃝ 2 susume ⃝
1 tomare ⃝ 3 sagare ⃝
4 migi ⃝
5 hidari
tance supposed with the specifications of the voice con-
trolled wheelchair though it can expect that the braking
distance decreases by making improvement in the perfor-
mance of the laptop and the low running speed. Here, the
braking distance was about 50cm when button operation
is used without using the voice input.
6 sukoshi- ⃝
⃝ 7 sukoshi- ⃝8 sukoshi- ⃝ 9 sukoshi-
3.3 Running experiment in the corridor
susume sagare migi hidari
The next experiment is a running experiment. This ex-
Fig. 5 Mode illustrations. periment was carried out with same five persons of pre-
vious experiment. A running place was in the corridor
of campus, and the width of the corridor was about 2m,
4. Quit button: When a button is pushed, the control and one obstacle was put on the hall of the corridor. We
system is quit. set two courses A and B as shown in figure 8. The total
5. Control buttons: When one of the buttons is pushed, distances of course A and B were about 16m and 13m,
the wheelchair is moved. That is, a user can control respectively.
the system without voice command. Furthermore, The experimental running time, the number of basic
the present mode is highlighted. reaction command and the number of short moving reac-
tion command are shown in table 2. Because a little mov-
3. EXPERIMENT ing distance operation of the course B is more necessary
than the course A, there is more input of the short moving
3.1 Recognition experiment reaction command in the course B. Though the person E
To evaluate recognition performance of Julian, we ex- is one of the authors and he is control the system in a
periment speech recognition test with 15 students. The practiced operation, every person is almost running at the
target words are nine reaction commands and five verifi- same time.
cation commands as shown in table 1.
This experiment is carried on in laboratory room. 3.4 Running experiment in the room
There were some voices of other people in the recording The next experiment was carried out with the three
environment in the circumference. As the results, we ob- persons. A running place was in the room of campus. We
tained successful recognition rates of 98.3% of reaction presume the course which the start point is near the door
command and 97.0% of verification command, respec- and the destination point is one of the seats. The distance
tively. between the desks was about 160cm. The width of the
wheelchair was about 55cm. So, this experiment is a run-
3.2 Stopping experiment ning experiment in the narrow running course. To eval-
The action of this system executes until the next com- uate performance of our system, we experiment not only
mand is give. For example, a wheelchair goes straight voice operation but also the key operation of the laptop.
until stop or turn command is input. Then, we tested the The experimental running time, the running distance and
following three experiments to verify the operation of our the number of reaction are shown in table 3. The running
system. The overview of these experiments are shown in scenes of C are shown in the figure 10.
figure 6. Unlike the previous experiments in the corridor, the

- 338 -
 1000
stop line start line
900 goal
tomare
susume
800

ex.1 700

600
tomare course A
500

y [cm]
susume
course B
400
ex.2 obstacle
300

200
tomare
susume
100

ex.3 0
goal start
-100
-700 -600 -500 -400 -300 -200 -100 0 100 200 300
x [cm]
Fig. 6 Overview of stopping test.
Fig. 8 Test courses in the corridor.
100
Table 2 Experimental results of running experiment in
the corridor.
50
(a) Course A
distance [cm]

running the number of the number of

0
person time [s] basic command short moving command
A 166.14 12 7
-50
B 141.80 14 3
C 208.83 16 2
-100
A B C D E all D 217.26 16 6
person E 127.23 11 4
(a) ex.1 average 172.30 13.8 4.4
(b) Course B
300
running the number of the number of
person time [s] basic command short moving command
250
A 163.66 22 8
distance [cm]

B 177.43 16 8
200 C 158.68 17 6
D 185.46 9 15
150 E 156.33 14 10
average 168.30 15.6 9.4
100
A B C D E all
person

(b) ex.2 difference in running time by the user occurred. The per-
son C is one of the authors, and he controls the system in a
200 practiced operation. On the other hand, key operation was
almost the same running time as every person. Though it
150 had a difference individually, we confirmed that our sys-
distance [cm]

tem could run even through the narrow space, like the
100 room.

50 4. CONCLUSION
This paper developed a voice controlled wheelchair.
0
A B C D E all Three type commands, the basic reaction command, the
person short moving reaction command, and the verification
(c) ex.3 command, are given in our system. Speech recognition
used an open-source software Julian. We obtained a suc-
Fig. 7 Result errorbars of stopping experiments. cessful recognition rate 98.3%, 97.0% of nine reaction
commands and five verification commands, respectively.

- 339 -
 600
Goal A
B
500 C

400

300
y[cm]

200 (a) (b)

100

Start
0

−100
−600 −500 −400 −300 −200 −100 0 100

x[cm]

Fig. 9 The course and result path in the room.

(c) (d)
Table 3 Experimental results of running experiment in
the room.

(a) Voice control

running running the number of
person time [s] distance [cm] input command
A 143.95 1120 8
B 256.87 984 30
(e) (f)
C 49.43 959 7 Fig. 10 Running scenes.
average 150.08 1021 15
(b) Key control [4] Y. Matsumoto, T. Ino, and T. Ogasawara. Devel-
running running the number of opment of intelligent wheelchair system with face
person time [s] distance [cm] input command and gaze based interface. In IEEE Int. Workshop on
A 34.15 1042 7 Robot and Human Communication, pages 262–267,
B 49.70 947 19 2001.
C 38.51 983 9 [5] K. Choi, M. Sato, and Y. Koike. Consideration of
average 40.79 991 11.7 the embodiment of a new, human-centered inter-
face. IEICE Trans. Inf. & Syst., E89-D(6):1826–
1833, 2006.
The running experiment with some persons is carried out [6] K. H. Kim, H. K. Kim, J. S. Kim, W. Son, and S. Y.
in the campus corridor and room and the utility of our Lee. A biosignal-based human interface controlling
system is shown. a power-wheelchair for people with motor disabili-
Our system has not work out any countermeasure, ties. ETRI Journal, 28(1):111–114, 2006.
such as obstacle avoidance, yet. The future work is to [7] Y. Ichinose, M. Wakumoto, K. Honda, T. Azuma,
improve in our system with the practical use. and J. Satou. Human interface using a wireless
tongue-palate contact pressure sensor system and its
REFERENCES application to the control of an electric wheelchair.
IEICE Trans. Inf. & Syst., J86-D-II(2):364–367,
[1] G. Pires and U. Nunes. A wheelchair steered 2003.
through voice commands and assisted by a reactive [8] R. Barea, L. Boquete, M. Mazo, and E. Lopez.
fuzzy-logic controller. Journal of Intelligent and Wheelchair guidance strategies using eog. Journal
Robotic Systems, 34(3):301–314, 2002. of Intelligent and Robotic Systems, 34(3):279–299,
[2] K. Komiya, Y. Nakajima, M. Hashiba, 2002.
K. Kagekawa, and K. Kurosu. Test of running a [9] B. Rebsamen, C. L. Teo, Q. Zeng, M. H. Ang Jr.,
powered-wheelchair by voice commands in a nar- E. Burdet, C. Guan, H. Zhang, and C. Laugier. Con-
row space. JSME Journal (C), 69(688):210–217, trolling a wheelchair indoors using thought. IEEE
2003. (in Japanese). Intelligent Systems, 22(2):18–24, 2007.
[3] L.M. Bergasa, M. Mazo, A. Gardel, R. Barea, and [10] T. Kawahara and A. Lee. Open-source speech
L. Boquete. Commands generation by face move- recognition software julius. Journal of Japanese So-
ments applied to the guidance of a wheelchair for ciety for Artificial Intelligence, 20(1):41–49, 2005.
handicapped people. In ICPR, volume 4, pages (in Japanese).
4660–4663, 2000.

- 340 -
View publication stats