Real-time Control of a Mobile Robot Using
Electrooculography
Carlos Duque, Miguel Duarte, Marco A. Ribeiro, Sancho Oliveira, Anders Lyhne Christensen and Nuno Souto
Abstract—In this paper, we demonstrate how a human
computer interface (HCI) system based on electrooculography
(EOG) can be used to control a mobile robot. Such systems
can be particularly useful for people with limited mobility or
when the user needs to execute multiple tasks simultaneously.
We use an autonomic system in which high-level commands are
translated into context dependent actions by the robot. A total
of six different user inputs can be acquired and classified into
desired commands. We demonstrate our approach by controlling
a mobile robot that wirelessly receives high-level commands
such as “move forward” or “turn left at the next intersection”.
During our experiments, the users were watching a live video
feed captured by a camera onboard the robot, and were able to
simultaneously control it and execute other tasks, such as typing
or carrying out a conversation.
Keywords—Electrooculography, Mobile Robot, Human
Computer Interface, Real-time Control
I. I NTRODUCTION
E LECTROOCULOGRAPHY (EOG) is a technique used
to measure micro-potential signals produced by the
electrical difference between the retina (negative pole) and the
cornea (positive pole) of the eye. This technique allows the
acquisition of events, such as eye blinking or gaze direction,
which can be used as inputs for any given system. The use
of eye movement information enables the creation of a new
class of Human Computer Interfaces (HCIs) that can be used
to control electrical devices (e.g. wheelchairs) by people with
motor difficulties [1], [2] or to create new ways of interacting
with other systems like entertainment systems and augmented
reality systems [3], [4].
Autonomous robots have the potential to make some of our
everyday tasks easier. There are some tasks, however, that
require human decision or supervision. Robotic application
such as intelligent motorized wheelchairs, search and rescue,
or surveillance can benefit from high-level user input, such as
“explore” or “grab object”, while leaving low-level behavior,
such as locomotion or obstacle avoidance, to be autonomously
executed by the robot.
Assistive robotics and human computer interaction are
important research fields that are naturally connected.
This work was partly supported by FCT-Foundation of Science
and Technology under the grants SFRH/BD/76438/2011 and
SFRH/BD/87636/2012
All the authors are with Instituto de Telecomunicações, IT-IUL -
Edifı́cio II - Piso 0 Av. das Forças Armadas, 1649-026 Lisboa,
Portugal and with ISCTE-IUL, Av. das Forças Armadas, 1649-026
Lisboa, Portugal (carlos diogo duque@iscte.pt, miguel duarte@iscte.pt,
marco.ribeiro@lx.it.pt, sancho.oliveira@iscte.pt, anders.christensen@iscte.pt,
nuno.souto@lx.it.pt)
There have been an increasing number of projects whose
goal has been to improve the way people interact with
computerized systems. In the case of eye movement based
systems, cameras and video processing are usually used
to detect the eye movements, which require significant
computational resources [5]. Some of these systems depend
on head orientation and environmental factors such as light
conditions [6], [7]. There are other systems based on head
gestures, which are inadequate to be used by tetraplegic
people, who are unable to move their neck [8]. EOG is
independent from head orientation (it relies only on the
eye movement), from environmental factors such as lighting
conditions, and it does not require computationally complex
algorithms to distinguish between different events [9], [10].
In this paper, we show how EOG can be used to
remotely control a robot that has some level of autonomy
and intelligence. The robot uses high-level input from the
movement of the user’s eyes to select different behaviors,
such as “move forward” or “turn at the next intersection”.
These autonomous behaviors, along with obstacle avoidance,
allow the user to focus on different tasks while the robot
is navigating. The user only has to issue a command when
a decision is necessary, and the command can be given
without task switching since only eye movement is needed.
The proposed HCI uses an acquisition system, like the
one described in Section II, to acquire the user’s EOG
signals.Through eye movements, the user can control the
robot based on a video feed transmitted from the robot to
a workstation. The performed eye movement is translated
to a high-level command using the classification algorithm
described in Section III, and is wirelessly transmitted to
the robot which in turn translates the command to context
dependent actions as described in in Section IV.
II. ACQUISITION S YSTEM
To measure the potential differences that occur in response
to eye movements, a low cost electronic system was designed.
This system is composed by three stages with a total of seven
blocks (per channel) (see Fig. 1). A collection of Ag / AgCl
electrodes are placed over the skin at specific positions around
the eyes of a subject for signal acquisition. The two electrodes
marked with V+ and V- represent the vertical channel for
the acquisition of vertical eye movements such as look up
(LU), look down (LD), and eye blink (EB). The electrodes
marked with H+ and H- relate to the horizontal channel and are
responsible for the acquisition of horizontal eye movements,
namely, look left (LL) and look right (LR). The reference
electrode (REF) is placed on the earlobe.
The Amplification stage comprises three blocks per channel:
two instrumentation amplifiers, two high pass filters, and
two non-inverting amplifiers. The instrumentation amplifiers
are Analog Devices AD620, which are responsible for the
amplification of vertical and horizontal EOG signals. The
second block is composed of two passive fourth order high
pass filters with cutoff frequencies of 0.18 Hz, used to
eliminate the DC component of the vertical and horizontal
signals. The third block is a non-inverting amplifier with a gain
of 34.2 dB. The high pass filters were placed between the two
gain blocks, to avoid the high DC offset voltages involved and
system saturation. The overall gain of the system is 68.2 dB.
The Filtering stage is composed by two blocks per channel
with low pass filters and Twin-T notch filters. The low pass
filters are based on Sallen-Key topology (second order) with
a cutoff frequency of 33 Hz to eliminate high frequency
noise, although these low pass filters were not enough to
completely remove the 50 Hz powerline interference. After
several experiments, the best solution found to remove this
interference was to place two Twin-T notch filters after the
low pass filters.
Finally, the amplified and filtered EOG signals are
transferred to a chipKIT UNO32 prototyping platform. This
board has an embedded microcontroller (PIC32MX320F128H)
that contains twelve 10 bit analog to digital converters Fig. 1. EOG acquisition system composed of amplification, filtering and
(ADC) – only two out of the twelve available ADCs are used, analog to digital conversion stages.
one for each channel. We used a baudrate of 115200. The two
adder circuits are responsible for keeping the input voltages
of the ADCs between 0 V and 3.3 V. (NM) class to represent a state in which the user does
not perform any action. There are six events, but the CSP
III. C LASSIFICATION OF EOG SIGNALS plus LDA classification algorithm only have to distinguish
The microcontroller classifies the acquired signals in between five classes, so we implemented ten sub-classifiers.
real-time using a generalization of the Common Spatial Pattern Since the training procedure is computationally complex, it is
(CSP) plus Linear Discriminant Analysis (LDA) algorithm done in offline mode. The resulting CSP and LDA matrices
presented in [11], that classified EOG signals successfully in obtained in the training procedure are then programmed in
offline mode. In our case, we wanted to perform real-time the microcontroller in order to perform real-time classification.
classification, so we generalized the algorithm to perform a The training dataset was composed of data acquired from six
sample-by-sample classification. A sample is composed of different subjects.
two values, related to the vertical and horizontal channels.
A decision is returned when a certain number of consecutive IV. C ONTROLLING A ROBOT U SING E YE M OTION
samples are classified in the same manner. We noticed that The HCI system was used to control a small robot. Since
the algorithm confused EBs and LUs, because the waveforms there are two EOG channels available (vertical and horizontal),
of these two events only differ in their time duration. Based it was possible to control the robot’s movements in a two
on the observation that the time series of LU trials are always dimensional space. To control the robot, the UNO32 board
greater than the time series of EBs, it is possible to distinguish must send the classified EOG signals through USB protocol
these two classes. We introduced a new class named “Look to a workstation running a Java application that received the
Forward” (LF), that represents the union of LUs and EBs and device commands.
the classification algorithm is trained to make its classification. We used the LEGO Mindstorms NXT robotic platform for
So, to distinguish between LU and EB, the algorithm counts our tests. The robot was equipped with 3 ultrasonic sensors
the number of samples in a row that are classified as LF on the front (at −45◦ , 0◦ and 45◦ ) that were able to detect
and then if that number is between two predefined limits obstacles up to a maximum of 2.5 m. For locomotion, two
(BLINK MIN and BLINK MAX) it will return the EB action. motors with tracks were used. A small Viliv N5 computer was
On the other hand, if the value is above BLINK MAX the placed on top of the robot. The Viliv computer broadcasted a
algorithm will return LU. We also used the “No Movement” video feed to the user and transmitted commands from the user
to the robot. The Viliv computer was connected to the LEGO left on its vertical axis (outdoor mode).
NXT via USB and communicated with the user’s workstation • TURN RIGHT: the robot turns left at the next intersection
over a Wi-Fi network. The user, who operates the robot from a (indoor mode); the robot stops and then turns 45◦ to the
workstation, could see the live video feed sent from the robot. right on its vertical axis (outdoor mode).
In order for the user to decide which eye movements • MOVE FORWARD: the robot starts moving forward.
should be translated into commands, each command had • MOVE BACKWARD: the robot starts moving
to be preceded by blinking (EB command). After blinking backwards.
his eyes, the user had one second to use any of the other
Fig. 3 represents the plan of the floor of Instituto de
eye movements to issue a new command; otherwise eye
Telecomunicações-IUL and one of the routes used to test the
movements would be ignored. Our control system featured
robot. The user was sitting at a desk in front of the workstation
two distinct control modes: indoor mode and outdoor mode.
with the electrodes attached to his face. In the monitor, a
In the indoor mode, the robot was able to navigate indoor
live video was displayed, which was being sent by the Viliv
environment with minimal input from the user. A “move
computer onboard the robot that was quiescent in another
forward” command, for instance, would move the robot along
room. The initial state for the robot was always STOP. The
a corridor or an open space while avoiding obstacles. This
initial mode was the outdoor mode, since the environment was
mode also allowed the user to issue a “turn” command, even
if the robot was not near a intersection. In such cases, the robot
would continue following the corridor until it encountered
a intersection, which it would then navigate based on the
direction chosen by the user. The robot used the data provided
by the ultrasonic sensors to detect the walls and intersections.
The second control mode allowed the robot to navigate an
outdoor (or open-space) environment. Instead of turning at the
next intersection, the “turn behaviors” rotated the robot 45◦ on
its vertical axis in either direction. The robot maintained all
obstacle avoidance behaviors in both modes. The user could
switch between the two modes by blinking 3 times in a row.
The system featured acoustic signals to notify the user
of the current state, such as “waiting for command”. When
a command was issued, certain sounds would be played to
inform the user of which command the robot was executing.
By moving his eyes, the user sent commands to the robot.
Depending on the command received, the robot changed its
state according to the diagram shown in Fig. 2.
The following states were programmed:
• STOP: the robot stops and stays still in the same place.
• TURN LEFT: the robot turns left at the next intersection
(indoor mode); the robot stops and then turns 45◦ to the

Fig. 3. Plan of the floor where the robot control application was tested. On
Fig. 2. State machine diagram. The robot changes its state depending on the figure, it is possible to see one of the experimental routes used to test
the user eye movements. Before any state transition, the user must perform a the application as well as the actions the user must perform in each point to
voluntary blink. This prevents involuntary state transactions. reach its destination.
very large compared to the size of the robot. In the figure, all
the necessary actions to complete the route are represented
next to the black dots. To start the interaction, the user sends
the EB command, in order to be able to change the robot’s
state. After that, the user performs the LL action to make
the robot rotate 45◦ to the left on its vertical axis. Then an
EB, followed by an LU, is performed to make the robot start
moving forward.
To pass through the first door, the user corrects the robot’s
orientation by performing the LR action changing the state to
TURN 45◦ RIGHT. The user performs an LU to make the
robot start moving forward again. After the last black dot the
robot is moving forward, to stop it, the final action performed
by the user is LD, to change the state to STOP. Terminating
the video feed and closing the connection between the onboard
computer and the workstation concludes the interaction. Note
two important facts while using this HCI:
1) Every time the user wants to change the robot’s state,
first he needs to send the EB command. This prevents
normal eye movement from generating unintentional
inputs.
2) While the robot is moving, the user can always blink
once to enable state transition without affecting the
current movement.
The proposed HCI was tested with a single person. In
the open space mode, the robot behaved according to the
user’s commands, although some route adjustments were
required during the conducted experiments. Further tests
were conducted with the indoor mode. Small corridors were
assembled using wooden blocks. The robot was able to
correctly navigate the corridors. In these experiments, the user
reported a very good response by the robot, without the need
of making route adjustments as in the case of the open space
experiment.
V. C ONCLUSIONS
In this paper, we showed how an HCI system based on
EOG inputs can be used to control an autonomic mobile robot.
By having obstacle avoidance capabilities, the control of the
robot became easier for the user, since fewer commands had
to be issued. The robot also featured an indoor navigation
mode, which allowed the user to use even fewer commands
for corridor navigation. Using high-level commands such as
“turn left at the next intersection”, the user only had to decide
where the robot should go.
Different tests were performed using the two modes of
operation. Using the open space mode the user noted that
the robot could easily avoid obstacles, but even with this
functionality some route adjustments had to be made. Testing
the indoor, mode the user noticed a very good response by
the robot without the need of route adjustments, since the
robot always turned in an intersection according to the user’s
intention.
Although the tests of the proposed HCI were not extensive,
our classification system had been previously shown to work
successfully for different people [12]. Future work includes
testing our system with a larger number of people on
predefined navigation tasks in order to compare results and to
understand how different people interact with the developed
HCI. After these tests, we would ultimately like to test the
system in a motorized wheelchair. We believe this system has
the potential to improve the lives of motor-disabled people
and to help them become more independent in daily tasks.
Improvements could be introduced at the level of wheelchair
control, in the control of appliances, in domotic systems, or
even in the control of autonomic robotic arms.
R EFERENCES
[1] S. Yathunanthan, L. Chandrasena, A. Umakanthan, V. Vasuki, and
S. Munasinghe, “Controlling a wheelchair by use of EOG signal,”
in 4th International Conference on Information and Automation for
Sustainability (ICIAFS2008)., pp. 283 – 288, IEEE Press, Piscataway,
NJ, 2008.
[2] A. Al-Haddad, R. Sudirman, and C. Omar, “Guiding wheelchair
motion based on EOG signals using tangent bug algorithm,” in Third
International Conference on Computational Intelligence, Modelling and
Simulation (CIMSiM2011)., pp. 40 – 45, IEEE Press, Piscataway, NJ,
2011.
[3] M. Lin and B. Li, “A wireless EOG-based human computer interface,”
in 3rd International Conference on Biomedical Engineering and
Informatics (BMEI2010)., vol. 5, pp. 1794 – 1796, IEEE Press,
Piscataway, NJ, 2010.
[4] C.-C. Postelnicu, F. Girbacia, and D. Talaba, “EOG-based visual
navigation interface development,” Expert Systems with Applications.,
vol. 39, no. 12, pp. 10857 – 10866, 2012.
[5] D. Li, D. Winfield, and D. Parkhurst, “Starburst: A hybrid algorithm
for video-based eye tracking combining feature-based and model-based
approaches,” in Computer Vision and Pattern Recognition - Workshops,
2005. CVPR Workshops. IEEE Computer Society Conference on, p. 79,
IEEE Press, Piscataway, NJ, 2005.
[6] T. L. P. Jia, H. Hu and K. Yuan, “Head gesture recognition for hands-free
control of an intelligent wheelchair,” Journal of Industrial Robot, vol. 34,
no. 1, pp. 60–68, 2007.
[7] Y. Matsumoto, T. Ino, and T. Ogasawara, “Development of Intelligent
Wheelchair System with Face and Gaze Based Interface,” in Proc.
of 10th IEEE Int. Workshop on Robot and Human Communication
(ROMAN 2001), pp. 262–267, IEEE Press, Piscataway, NJ, 2001.
[8] Y. Kuno, S. Nakanishi, T. Murashima, N. Shimada, and Y. Shirai,
“Intelligent wheelchair based on the integration of human and
environment observations,” in Information Intelligence and Systems,
1999. Proceedings. 1999 International Conference on, pp. 342–349,
IEEE Press, Piscataway, NJ, 1999.
[9] R. Barea, L. Boquete, M. Mazo, and E. López, “Wheelchair guidance
strategies using EOG,” J. Intell. Robotics Syst., vol. 34, no. 3, pp. 279
– 299, 2002.
[10] A. Bulling, D. Roggen, and G. Tröster, “Wearable EOG goggles:
Seamless sensing and context-awareness in everyday environments,”
Journal of Ambient Intelligence and Smart Environments, vol. 1, no. 2,
pp. 157 – 171, 2009.
[11] C. Duque, M. Ribeiro, and N. Souto, “Multiclass electrooculography
using common spatial pattern,” in 35th International Conference on
Telecommunications and Signal Processing (TSP2012)., pp. 600 – 604,
IEEE Press, Piscataway, NJ, 2012.
[12] C. Duque, M. Ribeiro, and N. Souto, “A human computer interface
system based on electrooculography events for real time interaction,”
MSc thesis, Lisbon University Institute, Lisbon, 2012.