Professional Documents
Culture Documents
JEDThesis
JEDThesis
JEDThesis
BRAIN‐COMPUTER INTERFACING WITH EEG:
A LOOK AT EYE MOVEMENTS
Jordan E. DeLong
Submitted to the faculty of the Department of Psychological and Brain Sciences
in partial fulfillment of the requirements for the
Honors degree in Psychology
Indiana University
May, 2007
i | P a g e
Honors Thesis: Jordan DeLong
Accepted by the Psychological and Brain Sciences Faculty, Indiana University, in partial
fulfillment of the requirements for the Honors degree in Psychology.
Date of Oral Examination (May 3 , 2007)
ii | P a g e
Honors Thesis: Jordan DeLong
Acknowledgements:
Firstly, I’d like to thank Tom Busey for his exceptional guidance, mentorship, friendship and
inspiration over the past four years. My research experience as well as acceptance into graduate
school is a direct result of Tom’s influence during my undergraduate years.
I also need to thank my labmates that have made doing research an even more fun and
rewarding process. I want to single out Bethany Schneider (the lead author on my first
publication) and Dean Wyatte (whose programming expertise has been unbelievably valuable
and whose constant superiority at every game we’ve ever played has been unbelievably
aggravating). The Busey Lab has been the most constantly good part of my life these past four
years and I’m extremely grateful.
Finally, I’d like to thank my family for their love, support and funding. You guys have kept me on
track and helped me up when I’ve inevitably fallen.
Thank You.
iii | P a g e
Honors Thesis: Jordan DeLong
Abstract:
This study utilizes Independent Component Analysis to ocular artifacts in order to determine
their effects on an EEG based Brain‐Computer Interface (BCI). EEG data was recorded while four
subjects were visualizing movement to targets on the right and left of the chamber. The BCI was
tested and constructed using a combination of Fast Fourier Transform, Linear Discriminant
Analysis and Adaptive Boosting. Removing eye blinks increased classification of the system,
suggesting that the BCI is not driven by eye movements, but actually hindered by them. This
technique could be used in future designs to maximize categorization accuracy.
iv | P a g e
Honors Thesis: Jordan DeLong
Introduction
Brain‐computer interfacing (BCI) is a field of study that has been touted as a potentially
fertile area to develop technology that can be applied in medicine, electronics, entertainment,
and even the military. In its simplest form, a BCI is a device that takes neural signals and
converts them to digital signals that a computer can use for multiple purposes. This device can
be extremely useful and powerful, bridging the gap between the human body and the tools used
to interact with the environment. Instead of being forced to manipulate a real or symbolic
environment with body parts like hands and legs, a user should be able to interact with their
environment using the brain’s neural impulses.
The first mention of this type of device was proposed as a “carrier(s) of information in
man‐computer communication or for the purpose of controlling such external apparatus as a
prosthetic device or spaceship” (Vidal, 1973). This same paper states that this technology is
“potentially around the corner” in 1973, however BCI’s are not widespread or even well known.
Why is it that this simple concept is difficult to design?
The creation of a brain‐computer interfacing system requires expertise in an array of
disciplines, including neuroscience, medicine, computer science and mathematics, which have
only just started to join with an interdisciplinary focuses. While the emergence of Cognitive
Science in the 1970’s shows an academic turn towards the possibility of designing such a device,
many years are required for the necessary fields to advance before any usable prototype can be
crafted.
Another problem facing this field of study is that the public has already been presented
with limited futuristic images of brain‐computer interfacing that may make the best attempts
1 | P a g e
Honors Thesis: Jordan DeLong
look laughably bad. The best public example is in the film Star Wars: The Empire Strikes Back,
when Luke Skywalker is fitted with a robotic appendage that appears to perfectly mimic his old
hand down to pinprick reflexes. In popular fiction the idea of joining a brain and a computer
seems plausible, even trivial. This illusion that the brain and computer are easily compatible is
furthered by research in Artificial Intelligence that uses the modern personal computer as a
model for brain functioning. The reality of harnessing neural impulses is a much more difficult
operation that may never have as much success as presented in media and entertainment.
The first real issue in brain‐computer interfacing is engineering a way to capture
meaningful neural information. These issues have been addressed by the field of neuroimaging,
however different techniques require different analyses due to the methods of collection. Issues
of cost, portability, and acuity (temporal and spatial) must be optimized and traded off for a
successful system to work. While several options to advance this field exist, no clear alternative
has been touted as being correct.
The two main camps divided in the search for a workable BCI are North American and
European researchers. While the need and basic theory is similar in both camps, the main
difference is the proper neuroimaging technique to use. The main thrust of research in
American and Canadian institutions has been towards invasive techniques that require medical
intervention to acquire neural information from below the surface of the scalp. European
initiatives are more interdisciplinary, and tend to focus on non‐invasive techniques such as EEG
(Berger, 2006).
Invasive techniques face particular obstacles due to serious risks involved in acquiring
signals. To collect usable signals an electrode array is inserted into the cortex to collect
information from a small subset of neurons from each electrode. Inserting anything into cortex
2 | P a g e
Honors Thesis: Jordan DeLong
is medically complicated, and requires biocompatibility and a possibly for long‐term use. Leading
research at the University of Utah (Jones et al. 1992) and the University of Michigan
(Hoogerwerf and Wise 1994) have developed systems using silicon chips that can be
micromachined and can incorporate integrated electronics. The Utah Array has been
successfully used in certain patients such as Matt Nagle, a subject that is able to use move a
cursor in two dimensions around a screen (Hochberg et al., 2006). To create a safe cortical
implant new manufacturing techniques had to be developed to build this device (Jones,
Campbell et al. 1992).
This type of signal acquisition allows for adequate temporal and spatial acuity; however
the hazards of implanting a chip into the human brain cannot be underestimated. The long‐term
effects of implantation are not known, however the risk of infection during brain surgery can
prove fatal. Also, the ability to continuously record signal over time may be tainted by an
accumulation of dead cells around the electrode, making the signal useless (Wolpow et al.,
2000).
New ideas have sprung up when trying to solve this issue. New electrodes are being
prototyped that may have less negative effects on surrounding materials. An electrode array
also may telescope, allowing for the recording point to be extended closer to a neuron’s field.
Not safe for use in humans, animals are the primary test subjects for these developments. The
most stunning user of this technology has been videos showing successful trials of a restrained
monkey using a robotic arm from the lab of Miguel Nicolelis. These videos show the monkey
manipulating the arm to bring food to its mouth from multiple locations (Nicolelis, 2005)
It is also important to note other methods of signal capture such as
Electrocorticography, a method of localizing neural firing that involves an electrode web place
3 | P a g e
Honors Thesis: Jordan DeLong
under the scalp but on top of the scull. These are known to have few long‐term side effects and
don’t carry risk of brain infection. These webs use the same kind of analysis as the EEG, but
require less data cleaning because of more signal and less noise. A current ECoG web has shown
that control of a cursor using a joystick was at 90.2% accuracy while brain control was at 86.6%
accuracy after four days of training (Wilson et al., 2005).
Researchers in Europe have been focusing on techniques such as EEG that require little
to no surgical intervention. Instead of focusing on the output of individual groups of neurons,
EEG receives neural information in the form of electrical signals propagated through the scalp
from pyramidal cells. Useful information can be determined from the neural population in the
form of evoked potentials, slow cortical potentials and oscillatory EEG components (Guger et al.,
2000).
EEG frequencies have historically been divided into four frequency bands, Alpha (8‐13
Hz), Beta (14‐30 Hz), Theta (4‐7 Hz), and Delta (.5‐3 Hz). These frequencies are determined by
electrical fluctuations on a more global scale, and were primarily used in sleep research before
more in‐depth techniques were developed. Sleep research still investigates the contribution of
frequency from each of these bands and their correlation with sleep states. Since the advent of
computer analysis of frequency, more in‐depth frequency of analysis has shown promise in
building BCIs.
In the original paper by Vidal, the proposed method of analysis is to use frequency
analysis to differentiate between types of movement. This is all made possible through the use
of the Fast Fourier Transform (FFT), an analysis that takes a vector of data over a period of time
and returns the power of the signal along the frequency spectrum. The goal is to create a system
that can take the frequency data and correlate it to different intended movements.
4 | P a g e
Honors Thesis: Jordan DeLong
A promising way to create a BCI has been the use of the P300, a positive deflection wave
that can be used in an “oddball” paradigm to try and infer intention from activation. In order for
the P300 to be activated the signal needs a random sequence of events, a classification rule for
the sequence, and one class of stimuli that is presented less frequently. A simple task where a
6X6 matrix of letters is presented at random can trigger the “oddball” response when the
correct response is found (Donchin et al., 2000).
Systems using the P300 wave have been reviewed in‐depth and found to be effectively
moving at 5 characters per minute. While this system could provide binary yes/no decisions,
using it for any kind of motion would be too slow and disconnected. This type of input also
requires a great deal of concentration, and is less than accurate, picking the correct character
53% of the time. While this number is far greater than the 16.7% of the time the target is
presented, the P300 is less than efficient. Future additions to this system could be a “smart
speller” system that predicts words much like cellular telephones, as well as fewer choices to
improve accuracy (Vaughan et al., 2003).
Regardless of the known shortcomings of EEG, companies and research groups such as
the University of Graz have been working to create useful devices that will either emulate or
restore lost motor function. Instead of adding on a robotic limb Graz is working to adapt the
“Freehand” system that uses functional electrical stimulation (FES) of the unresponsive hand to
make muscles in the arm and hand move and grip objects using an outside power source. While
the originally designed input for an FES system is small muscle movements from the shoulder, a
functioning FES could take advantage of the living but inoperable tissue (Crago et al., 1996).
Another, slightly more optimistic approach of this technology is Emotiv systems, a
company that promises to enter the digital entertainment industry with systems that interact on
5 | P a g e
Honors Thesis: Jordan DeLong
“subconscious” levels using EEG technology. Through press releases and reviews, this company
claims that they have created a system that can infer movements such as raising, rotating, and
pushing objects. This same system claims to be able to capture facial expression and even how
hard the brain is focusing on a task (Emotiv, 2007). While claiming to “detect player thoughts,
feelings and expression”, the notion that any system can interface quite so readily with the brain
is mostly absurd.
Another system in development is the NeuroSky system, a single electrode attached to
headphones that promises to be an affordable method of collecting brainwaves and
eyemovement, including processing of emotion (NeuroSky, 2007). Claims to process brain and
eye signals are not fully plausible with a single electrode, due to the fact that cleaning a signal
requires more than one electrode. The contamination of eye movements to these systems
cannot be underestimated, and seems to be largely ignored by both of these companies.
Contamination of eye artifacts in the data comes from movements of the eyeball, eyelid,
and surrounding muscles as well as a refresh of the visual system. In most cases of large data
collection, trials of data that have been contaminated are simply left out of the sample. Omitting
data is less useful in BCI because of the noisy nature of the collected data due to a real‐world
environment. A possible solution is that data can be filtered using sophisticated computer
techniques such as independent component analysis to capture localization information, a
system used in epilepsy patients. Using these same techniques to filter information gained from
EEG based BCIs could significantly help filter out contaminated data before attempting to
process it.
These techniques work by finding components that underlie sets of random signals by
maximizing the difference between independent signals in a dimension. The strongest additive
6 | P a g e
Honors Thesis: Jordan DeLong
components are classified in sequence from most to least influential. Eye artifacts, being some
of the strongest that propagate through to the electrodes, can be separated from other data
readily and their influence removed from useful data. ICA is a useful and computationally
plausible system as long as it has enough data to produce meaningful results (Jung et al., 1998).
To determine how contaminating eye artifacts are to EEG‐based brain‐computer
interfacing a working sample must be constructed. This sample then can be examined with and
without the eye artifacts removed by ICA. The effect of these eye artifacts on EEG based BCIs
have not been addressed in the literature or from consumer companies. The effects of removing
these artifacts can either help or hinder the accuracy of categorization for a discrimination
system, with both effects describing what is driving the system.
If the BCI is being driven by eye movements, then removing them should lower
categorization accuracy. If the categorization accuracy is raised after cleaning out eye
movements then the system is being driven by something other than eye movements,
presumably neural impulses. Knowing what is affecting the system and how it can be improved
could be very useful in designing future systems.
7 | P a g e
Honors Thesis: Jordan DeLong
Methods
Participants
Five right‐handed Indiana University undergraduates (of whom two were male)
participated in this study. All reported normal or corrected‐to‐normal vision, and were
compensated for their participation. All were knowledgeable of the purpose and details of the
experiment. Human subject protocol was followed with an extension of Indiana University study
97‐381.
Apparatus
The EEG was sampled at 32 channels at 1000 Hz and amplified by a factor of 20,000
(Sensorium amps). Signal recording sites included a nose reference and during analysis
electrodes were restricted to (Figure 1). All channels had below 5‐kΩ impedance, and recording
was done inside a Faraday cage. Data was then filtered using a zero phase forward and reverse
digital FIR filter (Figures 2 and 3). Each trial was then individually processed to find power
spectral density using the Thompson multitaper method (Figure 4). A Sony CCTV camera and
monitor were also used to observe subject eye movements.
8 | P a g e
Honors Thesis: Jordan DeLong
Figure 1: Locations of all electrodes that were used. The reference electrode is in red
9 | P a g e
Honors Thesis: Jordan DeLong
Figure 2: An example of raw EEG data.
Figure 3: The same trial filtered by removing frequencies above 50 hz.
10 | P a g e
Honors Thesis: Jordan DeLong
Figure 4: The frequency distribution from 4‐42 hz for a sample data trial.
11 | P a g e
Honors Thesis: Jordan DeLong
Stimuli
In this experiment an arrow (pointing right or left) was presented to inform subjects
which direction to move. A tone was played to cue subject movement.
Experiment
This experiment aims to create a functioning EEG based BCI and determine its accuracy
with and without the presence of eye‐artifacts. Each subject underwent two sets of 100 trials.
During each trial subjects were shown a movement cue and told to visualize pressing one of the
two buttons either to the left or the right. After 800 ms a tone was played to cue subjects to
perform the movement they had visualized (sequence illustrated in Figure 5).
Data Analysis
The EEG data was recorded between the visual and aural cues (800 ms) and later
processed with a Thomson multitaper method to find the frequency content of the trial. To
determine the different signatures for both categories of movement a discrimination algorithm
must be implemented. While many guided learning algorithms could perform this task, the
simplest algorithm to implement is linear discriminant analysis (LDA). While different
implementations of LDA exist, this analysis utilizes the below equation (Fischer, 1936).
This algorithm uses the mean vector and covariance matrix for each category and
separates them with a linear boundary between them. While this solution works for a set of
12 | P a g e
Honorss Thesis: Jordan DeLong
Subject is shown EEG stopps
Trial cconcludes affter
movem ment cue. reecording. To
one is
sub bject pushes
EEG bbegins p
played (Subbject
lighhted button
n.
recording. cued to moove)
Fiigure 5: Sequeence of the exxperiment.
13 | P a g e
Honors Thesis: Jordan DeLong
data that has two classes, multi‐class systems require a more complicated system such as a
support vector machine (which will be revisited in the Discussion section).
Running the LDA creates a decision boundary for the two classes by finding the mean
and covariance for each class and comparing it to the sample. This boundary is independent for
each of the electrodes used to evaluate which direction the electrode reports the movement is
in. While an individual report can be made by an individual electrode, the system can be more
accurate by using an algorithm to take the input from each electrode and having them “vote”
for a winning direction. This algorithm is called Adaptive Boosting and is highly robust against
outliers and noisy data. For the purposes of this experiment the boosting algorithm has been
simplified (Figure 6) from the original implementation by Freund and Schapire (1999).
Taking the output from the boosting algorithm yields a decision based upon multiple
input sources. On certain occasions one electrode may be biased towards responding in favor of
a movement direction. To compensate for this kind of bias, the decision level can be
manipulated based upon the optimum level found in the training data. This helps optimize the
distribution of decision values. This can be seen as optimizing the signal to noise ratio for the
decision algorithm (Figures 7 and 8)
After the weak identifiers (electrodes) have been processed through the boosting
algorithm, the system can make a decision as to which of the two directions the subject was
planning to move. The overall accuracy of the system can then be gauged by comparing which
direction the subject was cued to go (after subjects were observed completing the task with
near perfect accuracy).
To determine the effects of eye artifacts on the analysis process a technique called
independent component analysis is used within a Matlab add‐on EEGLab.(Delorme and Makeig,
14 | P a g e
Honorss Thesis: Jordan DeLong
In
nput 1 Input 2
2 Input 3
ADA Boo
ost:
Att .5 decisionn level,
the systeem
c
concludes " "Red"
Fiigure 6: Diagrram of how thhe boosting a algorithm hass been strippeed. This algoriithm does nott
leearn by finding the predictiive value of eeach input oveer time.
15 | P a g e
Honors Thesis: Jordan DeLong
Figure 7: A incorrectly placed decision criterion creates more incorrect categorizations.
Figure 8: A centrally placed decision criterion will classify trials optimally.
16 | P a g e
Honors Thesis: Jordan DeLong
2004) to remove them. Six components were removed that corresponded to eye‐related
movement based upon the location and dispersion pattern. This can easily be done by visual
inspection of blink and eye movement trials collected before starting the experiment. Samples
of topographic scalp maps are in Figure 9 and 10. After processing, the data with eye
movements removed can be categorized in the same manner as unprocessed data and
compared for accuracy.
17 | P a g e
Honors Thesis: Jordan DeLong
Figure 9: A component comprising of strong ocular artifact interference.
Figure 10: A component based in the visual cortex and not due to ocular artifacts.
18 | P a g e
Honors Thesis: Jordan DeLong
Results:
Performance
Chance performance was placed at fifty percent due to the either‐or nature of the
decision task. The first round of processing entailed the categorization of data for the four
subjects that still contained eye movement components. After averaging all of the cross‐
validated categorization rates for each subject the average categorization rate were found to be
exactly at chance (50%).
After eye movements were removed using ICA the data were then categorized a second
time. The categorization rate showed that for each subject removing the eye artifacts improved
the performance of the system by 7.19 percent (Figures 11 and 12). A paired t‐test was used to
find that the difference was significant, t(15) = ‐2.3, p < .036. The mean of the group without
removal of artifacts had a lower categorization accuracy (M = 50.0, SD = 6.57) than the group
with eye artifacts removed (M = 57.2, SD = 11).
19 | P a g e
Honors Thesis: Jordan DeLong
Figure 11: Overall categorization rates for all subjects in both conditions compared.
20 | P a g e
Honors Thesis: Jordan DeLong
Figure 12: Mean Categorization for each subject before (blue) and after (green) eye artifact
removal.
21 | P a g e
Honors Thesis: Jordan DeLong
Discussion
The prospect of enhancing the performance of an EEG based brain‐computer interface
by removing eye artifacts through ICA could prove to be beneficial to professional development
of these systems. The NeuroSky and Emotiv systems are not equipped to utilize this kind of
system, however academic competitions that could support this processing have yielded
surprising results for analysis.
”BCI Competition III” was held in 2005, pitting different competitors with five different
datasets to come up with an analysis algorithm that would process the data most accurately.
The winner of the round of competition relating to this thesis, Yijun Wang of Tsinghua University
in Beijing (and collaborators Han Yuan, Dan Zhang, Xiaorong Gao, Zhiguang Zhang, and Shangkai
Gao), were able to categorize intended movement with an accuracy of 94.17%. Other
competitors were not as accurate, with some performing barely above chance (Figure 13).
Interestingly, these systems did not include a method of dealing with eye artifacts with
the exception of two competitors that removed all frequencies above and below 3 – 28 hz.
While the systems in “BCI Competition 2003” could possibly be improved by removal of eye
artifacts, there is little doubt that the system outlined in this paper could also be improved by
implementing strategies used in the competition. One major possible addition is the use of a
more sophisticated decision algorithm than LDA.
One such advanced decision algorithm is a Support Vector Machine, another linear
classifier that utilizes a hyperplane placed through multiple dimensions to differentiate
22 | P a g e
Honors Thesis: Jordan DeLong
Figure 13: The results of the 2003 competition with this study’s categorization rate placed for
comparison.
23 | P a g e
Honors Thesis: Jordan DeLong
between different classes of stimuli. Instead of running every electrode through its own decision
process all frequencies from all electrodes can be evaluated in multidimensional space
concurrently. While this technique is originally computationally expensive, after computing a
hyperplane that is able to reliably differentiate between classes of stimuli the only computation
required to find which class a sample belongs in is finding which side of the hyperplane its
coordinates lay on. A Support Vector Machine can also be extended into multiple classes, which
is a limitation of LDA systems. Due to the complexity of building the algorithm, a Support Vector
Machine was not utilized in this study.
Another possible addition to this project would be the full Adaptive Boosting algorithm
to prune out which electrodes were poor responders. Much like backpropagation in neural
networks, a full algorithm would take weights for each responding electrode and adjust them by
taking the dot product of the strength of the response, error and a learning rate. This system
was taken out of the processing routine because it required more trials for training ( > n = 100)
to work efficiently and because in the condition containing eye movements only frontal
electrodes were utilized. Time constraints also played a part in the decision to remove this part
of the algorithm.
Competitors in the 2003 competition implemented of bootstrapping to train systems
more effectively on a small amount of training data. This study could have utilized bootstrapping
in conjunction with ADABoost to train the system effectively by using repetitive categorization
of the data to find better information as to how categories are distributed overall.
A future direction of this research could be a to refine the discrimination algorithm using
several techniques above as well as finding better ways to remove the influence of eye artifacts
on data such as isolating frequency bands of eye movements and removing them in the FFT
24 | P a g e
Honors Thesis: Jordan DeLong
instead of using ICA. This would require isolating artifact frequencies, a task that could achieved
by isolating their components through ICA and finding the activity of the frequency spectrum.
This system may need to be tailored to the frequency of each subject’s eye artifacts.
Implementing this system requires only a few training trials for isolating ocular artifacts.
With these strategies it is a possibility that an EEG based interface could be used with
some degree of ease. Although mass adoption of this type of device is years off, coming up with
more efficient and accurate systems can only hasten their arrival.
25 | P a g e
Honors Thesis: Jordan DeLong
References:
Berger, T. (2006, July). Report on BCI Research in Europe. Talk presented at the WTEC BCI
Conference at the NSF, Washington, DC.
Crago P.E., Lan N., Veltink P.H., Abbas J.J., Kantor C.K. (1996) “New control strategies
forneuroprosthetic systems”, J. Rehab. Res. Dev., Vol. 33, No. 2, 158‐172.
Delorme, A. Makeig, S. "EEGLAB: an open source toolbox for analysis of single‐trial EEG
dynamics," Journal of Neuroscience Methods 134:9‐21 (2004)
Efron, Bradley. 1979. "Bootstrap Methods: Another Look at the Jackknife". The Annals of Of
Statistics 7 (1): 1–26.
Emotiv. Emotiv Home Page, http://www.emotiv.com, site accessed March 5, 2007
Fisher, R.A. The Use of Multiple Measurements in Taxonomic Problems. Annals of Eugenics, 7:
179‐188 (1936)
Leigh R. Hochberg; Mijail D. Serruya, Gerhard M. Friehs, Jon A. Mukand, Maryam Saleh,
Abraham H. Caplan, Almut Branner, David Chen, Richard D. Penn and John P. Donoghue (13 July
2006). "Neuronal ensemble control of prosthetic devices by a human with tetraplegia".
Nature 442: 164‐171. Retrieved on 2006‐09‐10.
Neurosky. Neurosky Home Page. http://www.neurosky.com, site accessed March 5,2007.
Nicolelis, MA (2003) Brain‐machine interfaces to restore motor function and probe neural
circuits. Nat Rev Neurosci. 4: 417‐422.
Schapire, RE. Singer, Y. Improved Boosting Algorithms Using Confidence‐Rated Predictors.
Machine Learning, 37(3):297‐‐336, 1999.
Vaughan TM, W. J. Heetderks, L.J. Trejo, W. Z.Rymer, M. Weinrich, M. M. Moore, A. Kubler, B.
H.Dobkin, N. Birbaumer, E. Donchin, E. W. Wolpaw, and J. R. Wolpaw. Brain‐computer
interfacetechnology: A review of the Second International Meeting, 2003.
Vidal, JJ., (1973). Direct brain‐computer communication, Ann. Rev. Biophys Bioeng, 2, 157‐158.
Wolpaw J.R., Birbaumer N., Heetderks W.J, McFarland D.J., Peckham P.H., Schalk G., Donchin E.,
Quatrano L.A., Robinson J.C. and Vaughan T.M. (2000a). Brain‐Computer Interface
Technology: A Review of the First International Meeting, IEEE Trans. Rehabil. Eng., 8,161‐
163.
26 | P a g e