290
1, JANUARY 2012
Tongue-Rudder: A Glossokinetic-Potential-Based
Tongue–Machine Interface
Yunjun Nam, Qibin Zhao, Andrzej Cichocki, Member, IEEE, and Seungjin Choi*, Member, IEEE
Abstract—Glossokinetic potentials (GKPs) are electric potential
responses generated by tongue movement. In this study, we use
these GKPs to automatically detect and estimate tongue positions,
and develop a tongue–machine interface. We show that a specific
configuration of electrode placement yields discriminative GKPs
that vary depending on the direction of the tongue. We develop a lin-
ear model to determine the direction of tongue from GKPs, where
we seek linear features that are robust to a baseline drift prob-
lem by maximizing the ratio of intertask covariance to intersession
covariance. We apply our method to the task of wheelchair con-
trol, developing a tongue–machine interface for wheelchair control,
referred to as tongue-rudder. A teeth clenching detection system,
using electromyography, was also implemented in the system in
order to assign teeth clenching as the stop command. Experiments
on off-line cursor control and online wheelchair control confirm
the unique advantages of our method, such as: 1) noninvasiveness,
2) fine controllability, and 3) ability to integrate with other EEG-
based interface systems.
Index Terms—Electric wheelchair control, glossokinetic poten-
tials (GKPs), tongue–machine interface.
I. INTRODUCTION
SSISTIVE technologies have been developed for persons
A with limb motor disabilities and even for those with com-
plete quadriplegia, in order to provide an alternative communi-
cation channel and help the persons in performing daily tasks
by utilizing biosignals measured from various sensors placed on
the persons. For several reasons, highlighted in [1], the tongue
Manuscript received December 28, 2010; revised April 22, 2011 and August
10, 2011; accepted October 4, 2011. Date of publication October 28, 2011; date
of current version December 21, 2011. This work was supported in part by the
National Research Foundation (NRF) of Korea under Grant 2011-0018283 and
Grant 2011-0018284, by the NRF World Class University Program under Grant
R31-10100, and by the TÜBİTAK under Project 110E232. A portion of this
work was carried out while Y. Nam was visiting the Advanced Brain Signal
Processing Lab, Brain Science Institute, RIKEN, Wako-shi, Japan. Asterisk in-
dicates corresponding author.
Y. Nam is with the School of Interdisciplinary Bioscience and Bioengineer-
ing, Pohang University of Science and Technology, Pohang 790-784, Korea
(e-mail: druid@postech.ac.kr).
Q. Zhao is with the Advance Brain Signal Processing Lab, Brain
Science Institute, RIKEN, Wako-shi, Saitama 351-0198, Japan (e-mail:
qbzhao@brain.riken.jp).
A. Cichocki is with the Advance Brain Signal Processing Lab, Brain Sci-
ence Institute, RIKEN, Saitama 351-0198, Japan, and also with the Systems
Research Institute, Polish Academy of Science, 01-447 Warszawa, Poland
(e-mail: a.cichocki@brain.riken.jp).
∗ S. Choi is with the Department of Computer Science and Division of
IT Convergence Engineering, Pohang University of Science and Technology,
Pohang 790-784, Korea (e-mail: seungjin@postech.ac.kr).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TBME.2011.2174058
0018-9294/$26.00 © 2011 IEEE
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 59, NO.
is regarded as a suitable organ for manipulating assistive de-
vices involving motor control. These reasons are as follows:
1) the tongue is directly connected to the brain by cranial nerves
and the distance from the brain is relatively short; 2) the tongue
generally escapes severe damage in spinal cord injuries, and
in many cases, it is more slowly affected than limbs of per-
sons suffering from most neuromuscular degenerative disorder;
3) the tongue consists of special muscles suitable for complex
vocalization and ingestion tasks so that it can move very quickly
and accurately with little fatigue; and 4) tongue movements can
be hidden by the mouth cavity, which is a cosmetic advantage.
Several tongue-operated assistive devices, which benefitted
from the aforementioned advantages, have been developed. The
“tongue drive” system uses a tongue-mounted permanent mag-
net and magnetic-sensors-implanted dental retainer for tracking
tongue movement [2]. The inductive tongue computer interface
(ITCI) observes changes in the inductance from coils (attached
to the palatal plate), which are caused by the movement of
ferromagnetic material (attached to the tongue) [3]. The “think-
a-move” system can recognize commands by analyzing acoustic
patterns generated by specific tongue motions such as flicking
of the tongue to the left/center/right gum line [4]. The “tongue-
mouse” uses piezoelectric ceramic materials, which produce
small charges when pressed, to sense a push by the tongue [5].
The “Tonguepoint” is a special mouthpiece, and it has a small
pressure-sensitive joystick on the hard palate region [6].
Most of the existing methods for tongue–machine interfaces
require that sensors be placed on the tongue or inside the mouth
in order to acquire signals involving tongue movement. They
are limited to only a few commands so that fine control of
device is not possible. In this paper, we present a novel tongue–
machine interface where we detect tongue positions solely by us-
ing glossokinetic potentials (GKPs) that are EEG signals related
to tongue movements. In contrast to existing tongue–computer
interface systems, our method does not require any extra sensors
placed inside mouth, which might be more comfortable to ac-
quire signals involving tongue movements. Moreover, as will be
shown, our method enables fine-grained device control with an
accuracy (in terms of difference between the estimated direction
and the true direction) of approximately 20◦ .
In our preliminary results reported in [7], we showed that a
specific configuration of electrode placement yields discrimi-
native GKPs that vary in accordance with the direction of the
tongue. Thus, GKP is a direct continuous interpretation of the
direction of the tongue, leading to fine-grained device control.
In this paper, we develop a linear model to associate GKP with
the direction of the tongue, where we seek linear features that
are robust to the baseline drift (the gradual and linear shift in
NAM et al.: TONGUE-RUDDER: A GLOSSOKINETIC-POTENTIAL-BASED TONGUE–MACHINE INTERFACE
Fig. 1. Overview of the “tongue-rudder” system. GKP is analyzed to estimate
the direction of the tongue (in the range between − π2 and π2 ) that facilitates
smooth control of the wheelchair. The electromyograph (EMG) signals that
arise when clenched are used for sending special commands such as “stop” or
“turn ON/OFF.”
the baseline over a specified period of time), by maximizing
the ratio of intertask covariance to intersession covariance. We
apply our method to the task of wheelchair control, developing
a tongue–machine interface for wheelchair control, referred to
as tongue-rudder.
The rest of this paper is organized as follows. In Section II, we
first briefly explain GKP that is a slow wave response caused by
tongue movement. We then describe seesaw-like GKPs that vary
in accordance with tongue positions that are observed under a
specific configuration of electrode placement. These seesaw-like
GKPs are one of the main topics discussed in this paper, because
they enable us to develop an EEG-based tongue–machine inter-
face. In Section III, we develop a method for interpreting GKP
in terms of the direction of the tongue. We describe a linear
feature extraction method that is robust to the baseline drift.
In Section IV, we describe the implementation details of our
tongue-rudder system, an overview of which is graphically de-
picted in Fig. 1. The tongue-based interface is similar in function
to a steering wheel and allows the wheelchair to turn smoothly.
The clenching-based interface is similar to a brake as it toggles
between run and stop states. Section V presents experiments on
off-line cursor control and online wheelchair control. Finally,
conclusions are drawn in Section VI.
II. GKPS
In this section, we briefly explain GKPs, and then, introduce
seesaw-like GKPs that are used in our system.
A. Background: GKPs
GKPs are electric potential response generated by tongue
movement [8], [9]. The tip of the tongue has a negative electric
charge with respect to the root. As a result, if the tongue touches
the palate or inside the mouth, a discharge is generated that
291
affects the potential levels on the scalp (see [8, pp. 114–117]).
Vanhatalo et al. [10] also reported that GKP can be removed
by insulating the surface of the tongue in order to block the
discharge. This GKP has been studied primarily within EEG
analysis. Since it originates from a noncerebral region, it can
interfere with the observation of brain activities. Thus, it has
been ignored or actively removed during the recording or the
preprocessing stages, likewise other artifacts such as EMG or
electrooculogram (EOG) [11].
GKP has attracted little attention, compared to EMG or EOG,
since GKP is easily regulated than EMG or EOG. First, GKP
can be consciously suppressed by the subject. Unlike EOG ar-
tifacts, which are caused by an involuntary eye–ball movement
or blinking of the eye, GKP is caused by a voluntary tongue
movement. Thus, it can easily be regulated unless the subject
has special symptoms such as Parkinson’s disease or tremor
disorder. Second, GKP can easily be removed by simple high-
pass filtering. EMG is also a consciously regulatable artifact
like GKP. However, once it is generated, it severely hampers
EEG analysis because it contaminates all frequency bands in
the millivolt scale [12]. In contrast, GKP has influence only
on low-frequency bands, so it induces less deterioration and can
easily be removed by simple high-pass filtering with a cutoff fre-
quency at 4 Hz. Because of these reasons, GKP has been studied
less and considered only when the EEG analysis is related to
vocalization or ingestion tasks [13], [14].
B. Seesaw-Like GKPs
During our GKP experiment, we observed the correlation
between GKP and horizontal tongue movements [see Fig. 2(a)],
with the specific electrode placement, as depicted in Fig. 2(b). In
other words, we observed that when the tongue touches the lips
and then moves in a horizontal direction, the dc levels of EEG
signals recorded from two earlobes move to opposite directions
and their magnitude is linearly proportional to the direction of
the tongue, as can be seen in Fig. 4.
To validate the seesaw-like patterns of this GKP, we per-
formed two experiments. In the first experiment, we investigated
the spatial pattern of GKP, while in the second experiment, we
clarified the continuous relationship between the direction of
the tongue and GKP.
For the experiments, EEG signals were recorded using a
g.USBamp (g.tec, Graz, Austria) device system with Ag/AgCl
electrodes. Channel position has already been indicated in
Fig. 2(b). This specific layout of electrodes was applied for
maximizing the sensitivity to antisymmetric potential changes.
To remove oscillations of higher frequency ranges, signals were
low-pass filtered with a cutoff frequency at 4 Hz by a Butter-
worth filter.
In the first experiment, three cues assigned to the left (L,
touching inside of the left cheek), front (F, touching lips on the
front side), and right (R, touching inside of the right cheek)
were sequentially shown to the subjects. These points (L, F, and
R) will be used as the referential positions in the entire paper.
In accordance with the cues, the subjects continuously moved
their tongue from L to F to R; this action is similar to that of
292 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 59, NO. 1, JANUARY 2012
Fig. 2. (a) If the tongue touches the inside of the cheek (buccal wall) by passing
in between the upper and the lower teeth, a potential alteration is generated. We
analyzed this potential to detect the direction of the tongue within the range
−π/2 (touching the inside of the left cheek) to π/2 (touching the inside of the
right cheek). (b) Channel position for EEG recording during the implementation
step. The reference electrode was mounted on top of the head (Cz, dashed circle),
and two ground electrodes were mounted on the forehead and at the back of
the head (POz, AFz, black dashed circle). Then, signals were recorded from
temporal regions (F7/8, T7/8, P7/8, and left/right earlobes (LE/RE), and black
circle) for measuring antisymmetric GKP. (a) Tongue movements during the
experiments. (b) Channel location.
licking the lips. They were also instructed not to move their chin
during the tongue movements. This was done in order to prevent
shifting or tilting of the electrodes. The length of each cue was
4 s and the single session consisting of three different cues was
repeated eight times.
In each session, we measured the potential differences while
the tongue moved from L to F and from F to R, and their mean
differences on each channel were visualized in Fig. 3 using the
EEGLAB topoplot function [15]. In both the cases of L to F
and F to R, the potential levels on the electrodes of the left
hemisphere increased, while the levels on the electrodes of right
hemisphere decreased. In addition, the absolute value of the
potential changes on the electrodes increased toward the lateral
direction. The potentials of the electrodes on the LE and the RE,
which are located closely to the cheek and the tongue, showed
larger differences.
The second experiment was carried out to show the con-
tinuous relationship between the direction of the tongue and
potential changes [7]. The subject was asked to move his tongue
to the direction where the cue was located while maintaining a
contact between the tongue and the inside of the cheek. At the
beginning of each trial, the cue appeared from the right side and
moved to the left side along a semicircular path. Then, the cue
was returned to the right side by the same path, as shown in
Fig. 4(a).
Signals recorded on the LE and the RE are shown in Fig. 4(b)
and (c). We selected these channels because they showed the
largest potential changes in the first experiment. We denoted
the signal from the LE by x1 (t) and the signal from the RE
by x2 (t). When the tongue was moving from the right side to
the left side (0–6 s), the signal of x1 (t) continuously decreased,
while x2 (t) increased in similar continuous manner. On the
other hand, when the tongue was moving to the right side (8–
14 s), x1 (t) increased while x2 (t) decreased. The potential
difference between the two channels [see Fig. 4(d)] was almost
linearly proportional to the direction of the tongue.
In these two experiments, we observed that GKP evoked by
horizontal tongue movements follows a seesaw-like behavior
such that when one side goes up the other side goes down, and
the lifted heights are proportional to the distance from the center.
Thus, we refer to this pattern as seesaw-like GKPs.
The origin of this seesaw-like phenomenon can be explained
by the electric field generated by the charge on the tongue.
The tip of the tongue has a negative charge with respect to the
root. As a result, when the tongue touches the inside of the
cheek, the charge decreases the potential on the skin near the
contact point. In addition, its effect is maximum at the con-
tact point and decreases with the distance from the point (see
[16, p. 294]). Thus, if the negatively charged tongue moves to the
left side, the potentials on the electrodes in the left hemisphere
will gradually decrease and the potentials on the electrodes in
the right hemisphere will increase proportionally, and vice versa,
as shown in Fig. 4.
Moreover, the pattern of the seesaw-like GKP was clearly
distinguishable from known EEG phenomena such as event-
related potential (ERP) and slow cortical potential (SCP). This
seesaw-like GKP has persistent dc potential shift, unlike various
rhythmic EEG activities (e.g., δ ∼ γ rhythms or a sleep spin-
dle). In addition, the scale of GKP had a range of several tens
of microvolts. In the results of the first experiment, the maxi-
mum voltage difference between L and R for each subject was
82.1 μV (F7 for subject 1), 25.8 μV (T8 for subject 2), and
41.2 μV (T7 for subject 3). By way of comparison, ERP ex-
hibits amplitudes ranging only between 2 and 20 μV (see [17, p.
639 and p. 643]). The antisymmetric spatial patterns represented
in Fig. 3 are the final distinctive feature of the seesaw-like GKP.
III. METHOD
A. Feature Extraction
As mentioned in the previous section, potential differences
between the electrodes in the left and right hemispheres are
linearly proportional to the direction of the tongue. These po-
tential differences can be used to detect the tongue’s position. As
shown in Fig. 4, simple subtraction between two channels could
be an acceptable choice, but for maximizing the discriminabil-
ity, we applied a feature extraction approach with multichannel
recording.
Fig. 5(a) shows EEG signals recorded during eight sessions
of tongue movements. A single session consisted of four kinds
of tasks, namely: relax, left (L), front (F), and right (R). During
L/F/R tasks, the tongue touched each position for 4 s. During
the relax state, the tongue moved to the center of the mouth
while remaining not in contact with any place inside the cheek
or the palate. Our objective is to obtain an 1-D feature that is
well-matched to tongue positions. In the case of the signals in
Fig. 5(a), the corresponding feature will have the shape of a
ladder with three steps (each step corresponding to L, F, and
R). This ladder shape will be repeated eight times for eight
sessions.
NAM et al.: TONGUE-RUDDER: A GLOSSOKINETIC-POTENTIAL-BASED TONGUE–MACHINE INTERFACE
Fig. 3. Potential differences for different positions of the tongue. Potential differences while the tongue moves from L to F are presented on the left side and
differences from F to R are presented on the right side by topographical plots. For enhancing the visibility, we applied different scale ranges for each plot. It can
be seen that the antisymmetric spatial patterns are common, but their intensities are varied. (a) Subject 1. (b) Subject 2. (c) Subject 3.
Fig. 4. EEG signals were measured during repetitive tongue movement from
left to right. The subject was asked to move his tongue in the direction where the
cue was located, while maintaining contact between tongue and the inside of the
cheek. At the beginning of each trial, the cue appeared from the right side and
moved to the left side along a semicircular path. The cue then returned to the
right side by the same path. Traveling from one side to the other side took 6 s.
(a) Visual cue. (b) x1 (t): the signal recorded on the LE. Its gradually decreasing
baseline was drawn in red to emphasize the pattern of the drift. (c) x2 (t): the
signal recorded on the RE. (d) Differential signal xd (t) = x1 (t) − x2 (t).
The simplest approach to maximizing the sensitivity to the
tongue’s position might be to use principle component analysis
(PCA). Given eight-channel EEG signals X ∈ R8×N , we define
its segment X LFR(s) recorded during tongue movement from
L and F to R on session s, and its finer slice X L (s) , X F (s) ,
X R (s) , each of which corresponds to a single task. Thus, for the
first session (s = 1), the vector u that maximizes the feature’s
sensitivity is determined by solving the following equation:
 
 = arg max u X L F R (1) X 
u L F R (1) u . (1)
u
In Fig. 5(b), we illustrate the projected feature u  X with
a blue line. For the first session, we can see that the pattern
of three discrete steps representing three tongue positions is
an evident. These value ranges should ideally be at the same
levels throughout all sessions, as represented by the green line.
However, from the plot, it can be seen that feature values are
slowly biasing to the negative side. This will drag the detection
result to the right side. We also measured the absolute errors
293
between the projection results (blue) and the expected results
(green), then plotted it using the red line. It can be seen that the
error is increasing over sessions.
This dragging effect is caused by the drift, a gradual shift of
the baseline. In Fig. 5(a), it can be seen that the baselines of
each channel are gradually changing with their own direction
and amount. Under the linear projection approach, the drift will
lead to accumulated bias in the detection results.
The dc potentials of the EEG can be shifted by various neu-
ronal and nonneural factors [18], [19]. For example, particular
mental tasks related to cognitive or language processing can gen-
erate changes in dc potential changes (see [17, pp. 643–646]).
However, the drift, which is the gradual baseline shift, is more
closely related to nonneuronal factors [18], such as electrode
polarization and the skin potential. On the surface of an elec-
trode, the ions of an electrolyte (the conductive gel in this case)
are attracted to the metallic electrode. If these ions accumulate
over time, the charges on the ions can result in the gradual dc
potential shift [19]. In addition, perspiration and sympathetic
skin response (also known as skin conductance) can also gener-
ate the potential shift by changing the impedance between the
electrodes and the skin (see [18, pp. 114–117]).
Generally, this drift can easily be removed with a high-pass
filter above 0.5 Hz. However, this approach is not suitable for our
system, because GKP and the drift have similar pattern of the dc
potential shift. If the high-pass filter is applied, it will neutralize
not only the drift, but also GKP. Moreover, under the high-pass
filter, the values of each channel will converge to 0, according to
its low cutoff frequency. In this case, the detection result would
converge to a specific direction regardless of the direction of
the tongue. This problem was also reported in an EOG-based
interface system [20]. Fortunately, the potential changes of GKP
are swifter than those of the drift and form a spatially structured
pattern. As a result, we can extract the signal component for
GKP from the original signals.
To separate the signal component of GKP, we designed a
new feature vector that is insensitive to the changes caused
by the drift, while being sensitive to the tongue’s positions. In
Fig. 5(c), we represented the signals from the LE and RE [see
Fig. 5(a)], recorded during the first three training sessions to
2-D space. Signals were low-pass filtered and downsampled to
8 Hz. The Horizontal axis represents the potential on the LE and
the vertical axis represents the potential on the RE.
Within each session, while the “TASK” is changing from left
to front to right, the points move to the lower right direction
294 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 59, NO. 1, JANUARY 2012

Fig. 5. (a) Signals from eight EEG channels on temporal regions are recorded and low-pass filtered below 8 Hz. Colored boxes below the plot show the labels
for the signals (black: relax, red: left, green: front, and blue: right). The signals were recorded during eight sessions. Within each session, signals from the left
side (LE, F7, T7, P7) and the right side (RE, F8, T8, P8) show antisymmetric potential change when the tongue is moving to the right. Each signal is showing
drift of baselines, especially for P7 and P8. (b) Projection result from the signal of (a) to a feature vector from ordinary PCA is plotted in blue. The green line is
the expected result when the projection results for the first session are maintained without drift. These expected results are set to μ L (1 ) , μ F (1 ) , and μ R (1 ) , the
feature’s mean values for the first session. The absolute error between the two lines is plotted in red. Note that projection results are slowly biasing to the negative
direction and errors are increasing. (c) Signals from LE and RE from a different experiment were plotted in 2-D scatter plot for explaining the patterns for GKP and
the drift. Each “x” mark is colored by a corresponding task (black: relax, red: left, green: front, and blue: right. Routes for session 1 are drawn by small arrows).
When the tongue moves from left (red) to front (green) and right (blue), corresponding points move in the direction of wT . This pattern is maintained during each
of the three sessions, but due to the drift, points gradually move in the direction of wS . We tried to obtain a novel feature vector that is sensitive to changes for
the direction of wT , while insensitive to changes for wS , by using simultaneous diagonalization. (d) A projection result using the novel feature vector is plotted.
Compared to the result of (c), it can be seen that the drift is reduced and the error has also been decreased.

(wT ). This direction can also be extracted by the previous PCA
approach. If there is no drift, regions for each task should remain
at the same positions, illustrated by the boundary for session 1.
However, because of the drift, while the “SESSION” continues
from 1 to 2 to 3, the points slowly drift to the lower left direction
(wS ).
If the new feature obtained by the projection w T X is sensitive
wT ΣS w. Therefore, a new projection vector should be able to
maximize wT ΣT w, while minimizing wT ΣS w.
The intertask covariance matrix reflects the distribution of
data points within each session and it can be calculated by the
sum of the covariance matrices from each session as follows:

S

to the changes across the direction of wT (corresponding to ΣT = cov(X L F R (s) ). (2)

GKP), but is insensitive to the wS (corresponding to the drift), s=1

it will be robust against the drift.

On the other hand, the intersession covariance matrix is ob-
To measure the patterns of the aforementioned two types of
tained from the distributions of data points across sessions for
changes, let us assume two covariance matrices. The intertask
a single task, to reflect the direction of the drift. To obtain the
covariance matrix ΣT reflects the distribution during the transi-
intersession covariance matrix, all signals corresponding to the
tion of tasks within each session and is governed by the patterns
single task should be concatenated as follows:
of GKP. On the other hand, the intersession covariance matrix
ΣS reflects the distribution during the transition of sessions for  
the same task, and it is dependent on the direction of the drift. For X L = X L (1) , . . . , X L (S ) (3)
 
the projection vector w, the feature’s sensitivity to each change X F = X F (1) , . . . , X F (S ) (4)
can be calculated by their projected variances by wT ΣT w and  
X R = X R (1) , . . . , X R (S ) . (5)
NAM et al.: TONGUE-RUDDER: A GLOSSOKINETIC-POTENTIAL-BASED TONGUE–MACHINE INTERFACE
Fig. 6. During the training procedure, (a) and (b) measured EEG signals (blue
line) were bandpass filtered (green line), then (c) feature values were obtained
by projecting the signals to the linear filter w . From the feature values, the
mean value of each time segment for left, front, and right (μ L , μ F , μ R ) was
calculated. From these values, (f) linear model that translates the feature value
of (c) to the direction of (e) was generated. Newly obtained feature value in (c)
was translated to the color in (f) with the same height. The height of (f) and the
direction of (e) are matched by the same color. Sum of variances of all channels
was also measured in (d) to calculate the threshold for detecting EMG signals
of clenching teeth.
Then, from these concatenated signals, the intersession co-
variance matrix is calculated as the sum of each covariance
matrix such that
ΣS = cov(X L ) + cov(X F ) + cov(X R ). (6)
Because we are trying to find the w that can maximize
wT ΣT w, while minimizing wT ΣS w, the problem can be
solved by the following simultaneous diagonalization:
⎧ T
⎨ W ΣT W = ΛT
W T ΣS W = ΛS (7)
⎩ were recorded. The example signals for a single session are
ΛT + ΛS = I
when ΛT and ΛS are the diagonal matrices and I is an identity
matrix. We use the dominant eigenvector corresponding to the
maximum eigenvalue for ΛT and the minimum eigenvalue for

ΛS as the feature vector w.
We trained w by only using the first three sessions of signals,
  X was plotted in Fig. 5(d) to
and the new projection result w
be compared with the result in Fig. 5(b). It can be seen that the
new method reduced the drift and stabilized the feature values
within the normal value range. As a consequence, the errors
between the projection and the expected results are reduced, as
shown with the red lines.
B. Direction Detection Model
Using the aforementioned projection vector w, we measured
  X. The next step was to associate
the projected signal z(t) = w
this feature value with the direction of the tongue. We trained
w using three different kinds of signals: X L , X F , and X R ,
295
so z(t) can also be segmented into z L (t), z F (t), and z R (t). If
their mean values are calculated as μL , μF , and μR , then these
values can be referential points for direction detection. For the
calculation of mean values, we only considered the signals from
1 to 4 s after the cue, as indicated by the gray regions in Fig. 6,
in order to ignore intertask transition states.
To translate the features ranging from μL to μR , to corre-
sponding angles ranging from −π/2 to π/2, we used the linear
model depicted in Fig. 6 [7]. To simplify the illustration, we
only plotted two channels of signals from the LE x1 (t) and
the RE x2 (t) in Fig. 6(a) and (b), respectively. Whenever the
tongue moved from the left side to the right side, the potentials
of the two signals discretely altered to the inverse direction,
and the corresponding features in Fig. 6(c) also discontinuously
increased from μL to μF to μR . Once the cue was given, poten-
tial alteration was immediately occurred, then stabilized to new
levels within 1 s. This result is similar to the results reportedly
obtained in other GKP experiments (See [10, Fig. 2] and [8, pp.
117]). As a result, we could assume that the response time of
the interface would be less than 1 s.
Since the relation between the potential differences and the
corresponding directions is linearly proportional, the direction
of the tongue θt for the feature value μt can be represented as
⎧
⎪ π μt − μF
⎪
⎪ − × , (μL ≤ μt < μF )
⎨ 2 μL − μF
θt = π μt − μ F (8)
⎪
⎪ × , (μF ≤ μt ≤ μR )
⎪
⎩ 2 μ R − μF
Error, etc.
IV. IMPLEMENTATION
In this section, we explain how we connected the earlier
detection model to the electric wheelchair to implement the
tongue-rudder interface. In addition, we describe how the teeth
clenching detection system was integrated with the interface,
resulting in the tongue-rudder becomes a fully independent ve-
hicle control system.
To train the detection model, the signals from four sessions
represented by the plots in Fig. 6. In addition to the three ordinary
cues (L, F, and R), the signals for teeth clenching (C) were also
recorded. The duration of each session was 16 s and the duration
of each intersession relax was 8 s. As a result, a single training
procedure took 88 s for four sessions and three intersessions
relaxation periods. By analyzing the signals, we obtained the
GKP-based and the EMG-based models, and then, used these
models to enable the subject to control the wheelchair in the
ways outlined next.
A. GKP-Based Steering Wheel
By analyzing GKP, the model could detect the direction of
the tongue within the range from −π/2 to π/2. Based on the
detection result, the wheelchair could advance to any direction
from fully left to fully right. For example, if the tongue touched
the position F, the wheelchair moved forward. If the tongue
touched the position L, the wheelchair rotated to the left in the
same position. In the same way, if the tongue touched the left
296 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 59, NO. 1, JANUARY 2012
corner of the mouth (where the upper lip meets the lower lip,
the detection result may correspond to −π/4), the wheelchair
steered to the left, analogous to turning the steering wheel to the
left in a moving car. Due to this fine-grained controllability, the
subjects could drive the wheelchair smoothly, even on curved
pathways, as will be shown in Section V.
B. EMG-Based Brake and Recalibrator
The final goal of this research was to build a practical interface
that can control the electric wheelchair. To fulfill this require-
ment, the stop command, corresponding to the brake in a car, is
essential not only for the user’s comfort, but also for safety rea-
sons. To implement this stop command, we associated braking
with the EMG for teeth clenching. During the teeth clenching,
the subject cannot use the tongue-based interface because the
teeth blocks contact between the tongue and the inside of the
cheek. Therefore, we disabled the tongue-based interface when
the clenching was detected.
In order to detect the EMG signals associated with teeth
clenching, we measured the sum of the variances of all chan-
nels, during the clenching tasks in the training procedure. Sub-
sequently, during the testing procedure, if the sum of variances
of the new signal exceeded this value by 20%, the system recog-
nized it as the stop command, as shown in Fig. 6(d), and stopped
the wheelchair immediately.
In addition, we assigned this clenching task to other com-
mands essential for practical wheelchair control, such as calling
a recalibration procedure and toggling wheelchair power. This
procedure is described by the flow chart in Fig. 7.
First, we assign contiguous EMG signals longer than 2 s to
the command that calls the recalibration procedure. By using
the simultaneous diagonalization approach, we decelerate the
accumulation of the drift, but the approach cannot eliminate
it completely. If the accumulation is considerable, it has to be
cleared by resetting the baselines of each channel. The recali-
bration procedure does this. When the recalibration procedure
is invoked, the already stationary wheelchair system asks the
subject to touch lips on the front side (assigned to F and 0◦ of
the detection result) for 2 s. During these 2 s, the new baselines
are calculated from the recorded signal’s mean values.
Second, if EMG signals are maintained for more than 4 s
(that is, the EMG signals are detected even in the recalibration
procedure), the system enters a sleep state. In this state, the
system stops the wheelchair and ignore commands except the
wake-up command that is invoked by 2 s of teeth clenching.
Thus, the subjects could freely move their tongues for other
actions such as speaking or swallowing, when they did not want
to control the wheelchair.
V. EXPERIMENTS
By combining the aforementioned techniques, we built a
tongue-based wheelchair control system. For evaluating the per-
formance of our system, we designed three types of experiments.
The first one was the “cue following” experiment that measured
the accuracy of the interface. The subjects were asked to control
the interface in order to follow the given direction cue as closely
Fig. 7. Flow chart for the wheelchair control procedure. Sleep state (left):
After the training procedure, the system is in the sleep state. To initialize the
driving state, the subjects need to clench their teeth for 2 s. Before driving,
the system will invoke the recalibration procedure to set the initial baselines.
After that, the system enters the driving state, and the wheelchair begins to
move. Driving state (right): During the driving state, the subjects can move
their tongue to turn the wheelchair to the intended direction. When the subjects
want to stop the wheelchair, they can clench their jaws. When the subjects feel
that they are losing control because of the accumulation of drift, they can clench
their teeth for 2 s to invoke the recalibration procedure. If the subjects want to
turn OFF or park the wheelchair, they can maintain the teeth clenching for two
additional seconds, even during the recalibration procedure, that is, a total of
4 s is required to turn OFF the wheelchair.
as possible. The second experiment was the “off-line track” ex-
periment. In this experiment, the subjects were asked to control
the direction of a moving cursor so that they followed a track
displayed on the screen. This experiment confirmed that the user
could turn the wheelchair to the intended direction. In the final
experiment, called the “online track” experiment, the interface
was connected to a real electric wheelchair system, and the user
was asked to follow the track marked on the ground, in order
to ascertain whether the interface could assist the subjects by
improving controllability of the wheelchair.
Subjects 1 and 2 were the trained subjects who participated
in the development from the early stages while subjects 3 and 4
were naive subjects who had only 2 h of instruction and practice
with regards to the experiment. All subjects were healthy males
and their ages ranged from 24 to 31 years old (mean value =
28.5).
A. Cue Following
This experiment was designed to evaluate the accuracy of the
interface. A set of random directions was sequentially given to
each subject who then moved his tongue to match the direction
of the interface to the given cue as closely as possible. In Fig. 8,
the given cue is plotted in blue, while the predicted direction
is plotted in green. For each session, the subjects repeated the
training and testing procedure. To measure the error of the in-
terface, the mean of the absolute errors between the cues and
NAM et al.: TONGUE-RUDDER: A GLOSSOKINETIC-POTENTIAL-BASED TONGUE–MACHINE INTERFACE
Fig. 8. Results for cue following experiments. (a) Subject 1, session 2.
(b) Subject 1, session 6.
TABLE I
PERFORMANCE RESULT FOR CUE FOLLOWING EXPERIMENT
the predicted results was measured for each session, as shown
in Table I. The mean of all errors for 24 experiments was 18.9◦ .
B. Off-Line Track
During this off-line track experiment, the subjects were asked
to control the cursor to follow a track on the screen. As the cursor
slowly moved forward, the subject could control its direction by
moving his tongue. We used the circuit map of the Toyota Grand
Prix track located in Long Beach, California, as track’s image.
The image of the circuit map presented on the screen was
12 cm × 8 cm in size. The cursor advanced at a speed of
1.5 mm/s. The length of the course for a single lap was measured
to be 35 cm. When the detection result was pointing fully left or
fully right, the cursor turned to the corresponding direction at a
speed of 10◦ /s. The video clip for this experiment is available at
http://mlg.postech.ac.kr/research/tongue.html.
The results of this experiment are depicted in Fig. 9. The
black line is the given circuit line that the subjects were asked to
follow, while the red line is the actual path taken by the subjects.
To evaluate the performance, we measured the erroneous regions
between the circuit line and the actual path as represented by
the green region in Fig. 9(b). Values for the measured area are
shown in Table II.
Subjects 1–3 used the general model that was explained in
Section IV, but subject 4 could not use the clenching action
to invoke the recalibration procedure. In his case, clenching
caused physical trembling of the electrodes, so the potential
levels became unstable during the recalibration procedure. As
an alternative, we provided him with a button that could invoke
the recalibration procedure.
From the results of Table II, it can be seen that performances
improved as the number of sessions increased. This is due to
an adaptation effect that arose when the subjects found new
experimental tips such as the best contact spot or the best way
to increase the contact area. Among the outcomes, the result for
297
Fig. 9. Results for off-line experiments. The black line is the given circuit line
that the subjects were asked to follow and the red line is the actual path that
was taken by the subjects. To evaluate the performances, the erroneous regions
between the circuit line and the actual path taken were measured (colored with
green in (b) as an example) and their areas were compared. (a) Circuit course. (b)
Measured erroneous regions. (c) Subject 1, session 2. (d) Subject 1, session 5.
(e) Subject 4, session 1. (f) Subject 4, session 5.
TABLE II
UNIT AREA OF ERRONEOUS REGION FOR OFF-LINE EXPERIMENT
the subject 4 was the most outstanding. From session 4 onward,
he controlled the direction not only by the position of the tongue,
but also by the number of touches. For example, when he wanted
a rapid turning to the left side, he repeatedly touched the inside
of his left cheek until the intended direction was reached. This
approach was highly suitable for him, and he could follow the
track very precisely, as shown in Fig. 9(f).
C. Online Track
Using the practice outlined in the previous section, the sub-
jects improved their ability to control, and subsequently, at-
tempted the online wheelchair activity depicted in Fig. 10. The
wheelchair system used in this study was originally developed
in [21], for the EEG-based brain–computer interface. The speed
of the forward movement of the wheelchair was 0.9 m/s. When
it turned to a single direction in the same position, it could turn
at a speed of 30◦ /s.
298 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 59, NO. 1, JANUARY 2012

Fig. 10. Real-time online experiment. (a) We marked a track having the same topology as the Long Beach circuit map on the ground. (b) Implemented tongue-
rudder system. Each subject was asked to maneuver the wheelchair that followed the track. Time records for checkpoints are presented in (c), while recorded
signals and analysis result are presented in (d). The top figure in (d) is actually recorded signals during the driving, and the middle figure is the predicted direction
of the tongue. If the tongue moves to the left side, it shows higher values and the wheelchair turns left faster, and vice versa. In the bottom figure, the sum of
variances for eight channels was plotted. The higher value means stronger teeth clenching. The driving began [(S) ∼ (1)] and finished [(9) ∼ (E)] with 2 s of teeth
clenching. When passing through the left turning curves [at (2), (5), and (7), marked with flags in (c)], the subject moved the tongue to the left side, then the
system successfully recognized it, shown as the higher values in the middle figure of (d). (a) The marked track for the experiment. (b) An actually implemented
wheelchair system. (c) Time records for checkpoints. (d) Recorded signal and analysis result.

We marked the same topology as the Long Beach circuit
map on the ground, then asked the subjects to follow the track.
The subjects were able to control and maneuver the wheelchair
successfully. A video clip of this experiment is available
at http://mlg.postech.ac.kr/research/tongue.html. The recorded
EEG signals and analysis results obtained while one subject was
manipulating the wheelchair are shown in Fig. 10(d).
VI. CONCLUSION
In this study, we identified the seesaw-like GKP that revealed
antisymmetric potential changes triggered by contact between
the tongue and the inside of a cheek. We showed that the anti-
symmetric behavior of GKP is linearly proportional to the di-
rection of the tongue under a specific configuration of electrode
placement.
By using this novel potential response, we developed a
tongue–machine interface for wheelchair control, referred to
as tongue-rudder. We also developed a linear detection model,
where we sought linear features that are robust to a baseline
drift problem by maximizing the ratio of intertask covariance
to intersession covariance. The EMG detection system was then
integrated into the system to facilitate the recognition of three
essential commands for wheelchair control. Experiments on off-
line cursor control and online wheelchair control confirmed the
following advantages of our method.
Tongue-rudder can detect the tongue’s position from outside
the mouth. Many tongue-related interface systems inject sensors
or magnets into the tongue or teeth, but injecting devices into
parts of the mouth can cause discomfort, and it may not be
hygienic. However, this interface only uses EEG signals that are
recorded from the scalp.
The developed interface can precisely distinguish between
different directions of the tongue. Conventional tongue-based
interfaces generate discretely separated commands (similar to
toggling switches), but our interface enables continuous analog
manipulation. This feature has been smoothly integrated with a
wheelchair system and enables the wheelchair to be effortlessly
maneuvered, in a manner similar to driving with a steering
wheel.
Another expected advantage is that the system can easily
be integrated with EEG-based brain–computer interface sys-
tems. The only major physical difference between our system
and conventional EEG systems lies in the positioning of the
ground/reference electrodes. Furthermore, the system uses a
low-frequency band for the analysis, so the interference with
ordinary EEG-based BCI Systems, such as motor imagery or
P300, could be negligible. The integrated approach would be
NAM et al.: TONGUE-RUDDER: A GLOSSOKINETIC-POTENTIAL-BASED TONGUE–MACHINE INTERFACE
beneficial for extending the recognition capacity of the inter-
face if the target subject still has motility of the tongue.
In spite of aforementioned advantages, there are a number
of challenges that remain to be overcome in order to realize a
practical and reliable interface. First of all, the feature that we
suggested to reduce the accumulation of drift cannot achieve
complete elimination. The most promising approach is to sep-
arate the signals into glossokinetic component and drift com-
ponent, then use the drift component to update the baselines.
Second, the combination of the tongue-based interface and the
clenching-based interface is incomplete. Test subject 4 could not
use the combination method because his clenching task causes
fluctuation of the baselines. This implies that individual anatom-
ical differences can result in inability to operate the interface. In
this section, we feel that further experiments with more subjects,
including paralyzed patients, are required before this interface
can be implemented for general use.
REFERENCES
[1] G. Krishnamurthy and M. Ghovanloo, “Tongue drive: A tongue operated
magnetic sensor based wireless assistive technology for people with severe
disabilities,” in Proc. IEEE Int. Symp. Circuits Syst. (ISCAS), Island of
Kos, Greece, 2006, pp. 5551–5554.
[2] X. Huo and M. Ghovanloo, “Using unconstrained tongue motion as an al-
ternative control mechanism for wheeled mobility,” IEEE Trans. Biomed.
Eng., vol. 56, no. 6, pp. 1719–1726, Jun. 2009.
[3] L. N. S. A. Struijk, “An inductive tongue computer interface for control
of computers and assistive devices,” IEEE Trans. Biomed. Eng., vol. 53,
no. 12, pp. 2594–2597, Dec. 2006.
[4] ThinkAMove. Introduction to think-a-move’s technology [On-
line]. Available: http://www.think-a-move.com/pdfs/Intro_to_TAM_
Technology.pdf.
[5] W. Nutt, C. Arlanch, S. Nigg, and G. Staufert, “Tongue-mouse for
quadriplegics,” J. Micromech. Microeng., vol. 8, pp. 155–157, 1998.
[6] C. Salem and S. Zhai, “An isometric tongue pointing device,” in Proc.
ACM Conf. Human Factors Comput. Syst. (CHI), 1997, pp. 538–359.
[7] Y. Nam, Q. Zhao, A. Cichocki, and S. Choi, “A tongue-machine interface:
Detection of tongue positions by glossokinetic potentials,” in Proc. Int.
Conf. Neural Inform. Process. (ICONIP), Sydney, Australia, 2010, pp. 34–
41.
[8] B. J. Fisch, Fisch and Spehlmann’s EEG Primer, 3rd ed. New York:
Elsevier, 1999.
[9] D. Klass and R. G. Bickford, “Glossokinetic potentials appearing in the
electroencephalogram,” EEG Clin. Neurophysilol., vol. 12, p. 239, 1960.
[10] S. Vanhatalo, A. Dewaraja, M. D. Holmes, and J. W. Miller, “Topography
and elimination of slow EEG responses related to tongue movements,”
NeuroImage, vol. 20, pp. 1419–1423, 2003.
[11] J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T.
M. Vaughan, “Brain-computer interfaces for communication and control,”
Clin. Neurophysiol., vol. 113, pp. 767–791, 2002.
[12] M. Fatourechi, A. Bashashati, R. K. Ward, and G. E. Birch, “EMG and
EOG artifacts in brain computer interface systems: A survey,” Clin.
Neurophysiol., vol. 118, pp. 480–494, 2007.
[13] A. Ikeda, H. O. Lüders, R. C. Burgess, A. Sakamoto, G. H. Klem, I. H.
H. Morris, and H. Shibasaki, “Generator locations of movement-related
potentials with tongue protrusions and vocalizations: Subdural recording
in human,” EEG Clin. Neurophysiol., vol. 96, pp. 310–328, 1995.
[14] R. C. Pratap, “The speech evoked potential in normal subjects and patients
with cerebral hemispheric lesions,” Clin. Neurol. Neurosurg., vol. 89,
pp. 237–242, 1987.
[15] A. Delorme and S. Makeig, “EEGLAB: An open source toolbox for anal-
ysis of single-trial EEG dynamics,” J. Neurosci. Methods, vol. 134, pp. 9–
21, 2004.
[16] F. S. Tyner, J. R. Knott, and W. B. Mayer, Fundamentals of EEG Technol-
ogy: Vol. 1: Basic Concepts and Methods. Baltimore, Maryland: Lippin-
cott Williams & Wilkins, 1983.
[17] E. Niedermeyer and F. H. Lopes da Silva, Electroencephalography: Basic
Principles, Clinical Applications, and Related Fields. Baltimore, Mary-
land: Lippincott Williams & Wilkins, 1999.
299
[18] P. Tallgren, “Dc-EEG for routine clinical use: Methods and clinical im-
pact,” Ph.D. dissertation, Helsinki Univ. of Technol., Helsinki, Finland,
Dec. 2006.
[19] P. Tallgren, S. Vanhatalo, K. Kaila, and J. Voipio, “Evaluation of commer-
cially available electrodes and gels for recording of slow EEG potentials,”
Clin. Neurophysiol., vol. 116, no. 4, pp. 799–806, 2005.
[20] T. Yagi, Y. Kuno, K. Koga, and T. Mukai, “Drifting and blinking compen-
sation in electro-oculography (EOG) eye-gaze interface,” in Proc. IEEE
Int. Conf. Syst., Man, Cybern., Taipei, Taiwan, 2006, pp. 3222–3226.
[21] K. Choi and A. Cichocki, “Control of a wheelchair by motor imagery
in real time,” in Proc. 9th Int. Conf. Intelligent Data Eng. Automated
Learning (IDEAL), Daejeon, Korea, 2008, pp. 330–337.
Yunjun Nam received the B.Sc. degree in electrical
and computer engineering from Hanyang University,
Seoul, Korea, in 2008. He is currently working toward
the Ph.D. degree at the School of Interdisciplinary
Bioscience and Bioengineering, Pohang University
of Science and Technology, Pohang, Korea.
His current research interests include brain–
computer interface and electroencephalography
signal processing.
Qibin Zhao was born in 1979. He received the Ph.D.
degree in engineering from the Department of Com-
puter Science and Engineering, Shanghai Jiao Tong
University, Shanghai, China, in 2009.
He is currently a Research Scientist at the Labo-
ratory for Advanced Brain Signal Processing, Brain
Science Institute, RIKEN, Wako, Japan, where he is
a Visiting Research Scientist at Non-invasive BMI
Unit of BSI-TOYOTA Collaboration Center. His cur-
rent research interests include multiway data analysis,
brain–computer interface, and machine learning.
Andrzej Cichocki (M’95) received the Ph.D. and
Dr.Sc. (Habilitation) degrees in electrical engineer-
ing from Warsaw University of Technology, Warsaw,
Poland.
He is currently the Senior Team Leader and the
Head of the Laboratory for Advanced Brain Signal
Processing, Brain Science Institute, RIKEN, Wako,
Japan. He is the coauthor of more than 250 technical
papers and four monographs (two of them translated
to Chinese). He has been invited to the Council of
Canadian Academies Survey of Science and Tech-
nology Strengths as an author and coauthor of one of the top 1% most highly
cited papers in his field worldwide.
Seungjin Choi (M’93) received the B.S. and M.S. de-
grees in electrical engineering from Seoul National
University, Seoul, Korea, in 1987 and 1989, respec-
tively, and the Ph.D. degree in electrical engineer-
ing from the University of Notre Dame, South Bend,
Indiana, in 1996.
In 1997, he was with the Laboratory for Artificial
Brain Systems, RIKEN, Wako, Japan. From 1997 to
2000, he was an Assistant Professor in the School
of Electrical and Electronics Engineering, Chungbuk
National University. He is currently a Professor of
computer science at Pohang University of Science and Technology, Pohang,
Korea. His current research interests include machine learning, Bayesian infer-
ence, and probabilistic models.