»Mind-reading« with EEG-phase »decoding« (theta phase demodulation)
of brain working memory (WM) and decision-making processes
ABSTRACT
Various neuropsychological phenomena are commonly explored by electrophysiological methods (electroencephalography- EEG etc.), so the they can be explored on the ms level of electrical ac-
tivity of the brain. This allows the study of basic mechanisms of information processing. Interregional synchronized (phase-coupled, coherent) oscillatory activity is one possible mechanism of the
functional integration of different brain areas. Reports have shown that performing WM tasks causes an elevated rhythmic coupling in different areas of the brain; it has been suggested that this
indicates information exchange. In networks governed by an oscillation, the phase at which a cell fires relative to the oscillation could also carry information. Thus, the oscillatory coupling between
brain networks might reflect exchange of phase-coded information. Since the information exchanged is encoded in brain waves and measurable by electroencephalography (EEG), it is reasonable
to assume that it can be extracted with an appropriate method. In our study we made an attempt to extract the information using an artificial neural network (ANN), which can be considered as
a stimulus-response model with a state observer. The EEG was recorded from three subjects while they performed a modified Sternberg task that required them to respond to each task with the
answer "true"or "false". The study revealed that a stimulus-response model can successfully be identified by observing phase-demodulated theta-band EEG signals 1 s prior to a subject's answer.
The results also showed that it was possible to predict the answers from the EEG signals with an average reliability of 75% for all the subjects. From this we concluded that it is possible to observe
the system states and thus predict the correct answer using the EEG signals as inputs. Many oscillatory models of WM support principles of phase coding and temporal segmentation. This provides
a potential explanation of why it is that the brain oscillates. It is temporal clocking system for a neural code designed for handling ordered units of information. The newest aspects on the brain
coding reveal that phase coding serves as a general coding scheme in the brain, also in other higher brain processes, motor control and behaviour.
Neural encoding vs. decoding
 Neural en-coding : systematic variation of neural activity with respect to a behavioural variable (e.g. stimu-
lus-response function); research options are single/multiple-cell firing rate recordings, corellations of activ-
ity etc. Standard statistics can be used.
 Neural de-coding : extracting the value of specific behav. variable from the neural activity (neural “popula-
tion vectors” calculated), which- after known encoding- makes it possible to predict behaviour.
Phase coding in the brain and synchronous oscillations
 neural oscillations (originate from mainly post-synaptic electrical potentials of pyramidal cortical neurons);
phase- specific time point relative to the oscillatory wave or time lag between two waves.
 brain’s functionality, cognition and behavior are based on distributed, parallel information processing in
the brain; information exchange among the neuronal populations, which are anatomically separated but in-
volved in specific function/task, is carried out by synchronizing/coupling their oscillatory activity- integral
aspect of the function of the brain.
 Emerging view of brain processes - reverberations of reentrant electrical activity in complex neural net-
works. Interregional synchronized (phase-coupled, coherent) oscillatory activity is one possible mechanism
of the functional integration- binding. History: visual cortex of monkeys, cats; binocular rivalry…unified
perception
 Multiple superimposed synchronized (coherent) oscillations in different frequency bands represent func-
tional neuronal correlates governing specific mental functions. The proposed role of brain oscillations is
switching neural networks between different functional states with activating or inhibiting proper neural
systems. Evidence about causality of this relationship is also based on some computational simulation neu-
ral models.
 In oscillatory networks, the phase at which a cell fires relative to the oscillation could also carry informa-
tion. Thus, the oscillatory coupling between networks might reflect exchange of phase-coded information.
 information in the brain is carried by neuronal spikes or action potentials
 sensory inputs - cells fire, which aspects of the firing pattern are important, how the information is encoded
in theses spikes?
 The most accepted hypothesis - firing rates of spikes encode the information.
 BUT in networks governed by an oscillation, the phase at which a cell fires relative to the oscillation could
carry information.
 A network receiving the phase encoded firing patterns must receive the oscillatory signal as well in order
to utilize the phase code. Thus, the oscillatory coupling between brain networks often observed experimen-
tally might reflect exchange of phase coded information.
 temporal-coding hypothesis in humans- a general coding scheme in the brain?
This principle allows different neuronal populations in different regions to integrate firing patterns dependent on
the phase (shift)of the (theta) oscillatory drive input to the regions
 theta and delta oscillations facilitate phase coding and gamma oscillations help to decode combinations of
simultaneously active neurons.
 The timing of neuronal spikes in oscillatory networks is under the combined influence of external inputs
and the internal dynamics of the network. This shows how important are the internal dynamics of the brain
itself in producing representations.
The basic principle / model of phase coding in the
brain (Jensen, Lisman, 2005, 2006)
theta drive- a pacemaker, adjustable phase
The recurrent connections: sequence read-out or support repeated acti-
vation of the same set of memory representations
The inhibitory feedback : keeps the representations apart in time.
The phase decoder receives the phase coded information together with
the theta input. Neurons in the decoder fire when they receive input
from the encoder together with a depolarizing drive from the theta
rhythm.
When the phase difference is about 0 degrees the representation in the
middle of the list (C) is transferred.
When the encoder and decoder are in anti-phase information transfer
is blocked.
Thus it is possible to transfer different information from the phase en-
coder only by changing the phase relationship to the decoder.
Examples of phase coding in the brain
 theta phase precession in hippocampal “place cells”:
for cell participating in the representation of X, a gradual phase advance occurs over time- plausible expla-
nation for the theta phase advance observed in the rat hippocampus (episodic memory).
 Phase reset and memory:
a) 6–13 Hz oscillations stopping and restarting following both encoding and
retrieval - consistent phase following stimulus presentation- some stimulus-
evoked cognitive process resets the phase of ongoing oscillations. phase lock-
ing arises from an actual reset of ongoing oscillations (Rizzuto et al. 2003)
One possible role for phase reset would be to establish the synchrony
between disparate brain areas that may accompany cognitive processing?
b) Jensen and Lisman - phase reset to fit reaction time distributions in the
Sternberg task.
slow wave (4–12 Hz) phase reset to initiate serial scanning operations
after probe onset.
c) Oscillations as timing mechanisms for neural activity, organizing differ-
ent neural processes in different phases of an ongoing rhythm: phase-coding
of neural activity to separate the processes underlying memory encoding and
retrieval (Hasselmo et al., 2002; 2004).
Phase decoding/ “mind reading” at process/ content level
 The goal - how this oscillatory activity of the brain as revealed by the electroencephalography (EEG) or
magnetoencephalography (MEG) as well as more invasive recordings, is related to the dynamics of cogni-
tive performance, including working memory.
 Electrophysiological methods (electroencephalography- EEG, single or multiple cell recordings, etc) allow
the study of basic mechanisms of information processing in the brain. Excellent time resolution, lower spa-
tial resolution
 As information exchanged is encoded in brain waves and measurable by EEG, it can be extracted with an
appropriate method.
 Measure of synchronised brain oscillations: EEG- coherence
 Phase-decoding techniques, such as phase demodulation- possible to decode at least some of the informa-
tion exchanged and relevant to the tasks being performed. However, as the brain solves problems in parallel
not all measurable states are related to all responses. Therefore, only relevant states need to be extracted
from the measurements to predict the response.
Mind reading = brain reading
 Interdisciplinary approach: fMR, EEG, MEG, intracranial multiple electrode recordings etc.
 Prof. Dr. John-Dylan Haynes, Max Planck Institute: “Every thought can be related to characteristic
pattern of activation of the brain. After we teach the computer to recognize these patterns, it becomes
possible to read the mind of a person just from temporal-spatial patterns of his/ her brain activity.” ;
prof. Jack Gallant group etc. studies
 Research: predictions of intention/decisions, mental acts (attention, calculations, choosing), sensory
perception (photos), truth/lie decoders, motor acts (monkeys-screen cursor, robotic hands moving),
memories, episodic binar 0 /1 digital coding in hippocampus…
 Dreams? Conscioussness/ Coma, speech disorders?
 Need for high time and space resolution methods and combinations
WORKING MEMORY (WM)
 active role of short term memory – working memory- central for intelligent goal-directed behaviour, coher-
ent thoughts, language etc.
DEFINITION: complex of cognitive processes for time- and capacity- limited maintenance, manipulation and
utilization of mental representations
 specific multi-regional brain activity (fMRI), binding?
 (pre)frontal (VLFC, DLFC, AFC) cortex
 premotor cortex
 limbic cortex and other subcortical structures
 posterior association parietal areas
Many authors: increased rhythmic coupling (synchroniza-
tion, as measured by increased EEG coherence) of differ-
ent areas of the brain during working-memory tasks- re-
lates to information exchange or informational integration
Baddeley’s working memory model (Baddeley, 2000) (BINDING).
SIMON BREŽAN, MD
Uni Psychiatric Clinic Ljubljana, Ljubljana, Slovenia
E-mail: s_brezan@yahoo.com
Co-authors: Logar V & Belic A.
Faculty of Electrical Engineering, University of Ljubljana, Slovenia
 basic neurophysiological mechanism of WM: repeated reverberations of electrical impulses in reverbera-
tional (feedback) loops; repeated/posttetanic excitation of a synapse in excitational loop> increase of excita-
tory postsynaptic potential (EPSP) – postsynaptic facilitation- runs the STM circuit with no sensory input.
 dilemmas : how is information stored or manipulated in working memory? (reverberational excitatory
circuits, »delay-period activity« neurons) and what is the information stored ? (retrospective vs. prospec-
tive codes).
 elevated, persistent activity of the prefrontal cortex neurons, to guide forthcoming actions (Fuster, animal
studies), either from a brief sensory input or a readout from long-term memory.
 Memory processes are most closely related to theta and gamma brain rhythms.
Theta frequency- the active maintenance and recall of working-memory representations? an important role -the
phase characteristics of the theta rhythm The persistent activity of the prefrontal cortex neurons in the theta
rhythm carries the information about previously encountered stimuli or future responses required to solve work-
ing-memory tasks.
LIJ model of a network functioning as a multi-item short-term memory buffer.
Theta and gamma oscillations play an important role in the concept. An afterdepolarization (ADP) is triggered after a cell fires
(sensory input) and it causes depolarizing ramp that serves to trigger the same cell to fire again after delay. These ramps are
temporarily offset for different memories, an offset that causes different memories to fire in different gamma cycles. The key
function of this buffer is to perpetuate the firing of cells in a way that retains serial order. The repeat time is determined by
theta oscillations due to external input. Gamma oscillations arise from alternating global feedback inhibition and excitation
(the cell with most depolarized ramp will fire again) because of separate firing of different memory codes.
OUR RESEARCH
Objective and experimental design
Modified Sternberg paradigm of working memory
EEG and subject's answers were recorded during the modified Sternberg task
• data from 3 young, healthy, right-handed, male subjects.
• 29 scalp and 3 peri-ocular electrode channels were recorded for each subject.
• When the target stimulus was presented the participants had to reply by a button press answering whether
the probe was on a specific position in a sequence according to the spec.task
Aim
 brain as a non-linear dynamical system that can respond to multiple external stimuli with multiple respons-
es in parallel. We also view it as a causal and deterministic system, since in the experiment reported here,
responses occur after the presentation of stimuli and elicit similar responses
 we extracted the information using artificial neural network (ANN), which can be considered a stimulus-
response model with a state observer.
 The EEG was recorded from three subjects while they performed a modified Sternberg task that required
them to respond to each task with the answer "true"or "false".
 there should be a possibility to detect and to decode the exchanged information, not only with implanted
electrodes but also at a more macroscopic level.
 As the concept of phase lags is very similar to the phase modulation of the signals, where the phase shift of
the carrier wave codes certain information, we decided to build a model based on the phase-demodulation
approach of signal preprocessing
Logar et al. / Neural Networks 21 (2008) 881–887; Neurosci Res., 2008
METHODS
ANN and learning
• verification and validation three-layer feed-forward network (10-2-1)
• tangens sigmoid activation function of neurons on first and second layer
• linear activation function of neurons on third layer
• 15 principal components for the input and recorded answers for the output signal
• trained with scaled conjugate backpropagation algorithm
• Successful training of the ANN would support the assumption that the working-memory content encoded in
the EEG signals can be successfully extracted using an ANN.
Signal preprocessing of all electrodes
• simplistic filtering methods, for extracting the appropriate brain rhythm; (4Hz - 7Hz)
• the phase-demodulation method, for extracting the phase-coded information
• PCA- principal component analysis, for achieving the linear independence and detecting the source of the
signal; 5 signals instead of 29
• all this steps lead to more efficient training of the ANN
• the original data (29 electrodes) can be reduced to a small number, in this study 5, of the most significant prin-
cipal components, which contain 95% of the signal’s information.
Schematic representation of the model for response prediction from the EEG signals
Phase demodulation
 since the phase characteristics of the signals are supposedly playing an important role ininformation ex-
change , the signals were phase demodulated. to form the fourth input data set.
 Phase modulation is a method that modulates the transmitted information or signal as a variation of the car-
rier-wave phase. (carrier experimentally chosen- theta)
 The phase demodulation was calculated with the demod function in Matlab, which uses the Hilbert transfor-
mation for the calculations- just the reverse transformation to get the phase content of the signal.
 An example of phase modulation. diagram shows the modulating signal superimposed on the carrier wave.
The bottom diagram shows the resulting phase-modulated signal (output)
 Raw signals or other frequency bands or non- phase demodulated signals did not prove efficient in training
and prediction of ANN response.
RESULTS of ANN response prediction
1. raw EEG signal- no success
2. alpha band-pass filtered signals- no success
3. theta band-pass filtered signals – no success
4. Theta phase-demodulated signal: cca. 80% success in answer prediction- true or false
(similar results for all subjects)
Interpretation of results
 The study revealed that a stimulus-response model can successfully be identified by observing phase-demod-
ulated theta-band EEG signals 1 s prior to a subject's answer.
 The results also showed that it was possible to predict the answers from the EEG signals with an average
reliability of 75% for all the subjects.
 From this we concluded that it is possible to observe the system states and thus predict the correct answer
using the EEG signals as inputs.
 Considering all the simplifications that were made in the proposed model, taking into account that the brain
is a permanently adaptive system, and that some correct answers might have been guessed, the prediction
success of the ANN is very high.
 75% reliability of the ANN is high enough to suggest that the response prediction cannot be considered as a
random event. Also, the results are comparable for all three test subjects (cross- validation)
 Considering the functional anatomy of the brain and the origin of the EEG signals, it is remarkable that the
simplistic assumption of modulated EM sine waves achieves a reasonably good relation between the EEG
data and the subject’s performance.
 Thus, phase demodulation can be considered as a dynamic transformation of the signals that computes the
state estimates from the filtered EEG signals
DISCUSSION
Decoding of WM probe scanning/comparation, decision-making processes or motor activity?
 The calculated states’ estimates in the one-second interval prior to the response can be used to calculate the
subjects’ response. This suggests that the response signal is generated in the brain at least one second prior
to the response & shows that EEG signals are indicative of the brain state estimates that are relevant to the
stimulus response.
 Since the brain is a very complex system it is very difficult to say whether the trained ANN represents a
model of working memory for the logical or the physical answer.
 The Sternberg task elicits the preparation of motor activity with delayed execution, so the relation to the mo-
toric activity of the hand is obvious. However, if we consider the fact that the relation exists for the whole
second before the answer and theta freq. involvement, this suggests that the working memory is very likely
involved in the process.
 Free will? BUT as Libet's experiment showed the brain activity appears before conscious, voluntary inten-
tion to act. This indicates that we never consciously initiate our actions or decisions, even though our feeling
tells us differently. According to Libet, we can only influence actions by vetoing them.
Conclusions
 phase shifts of the EEG signals seem to have a key role during activity and information transfer in the
brain, while the phase-demodulation method proved to be a crucial step in the signal processing.
 it is well known that binding of the brain areas results in higher coherence values of the signals obtained
from these areas (Classen et al., 1998). Therefore, greater phase locking of the signals as a result of
binding is obviously the key to easier signal decoding when using the phase demodulation approach.
 phase demodulation and PCA give satisfactory results when predicting subject's answers: cca. 80%
reliability- posibility of using an artificial neural network for solving tasks with Yes/No response.
 Phase demodulation may thus be a useful approach for analysing the EEG related to working-mem-
ory. It is possible, however, that phase demodulation also describes some aspects of working-memory
activity in the brain itself?
 With the proposed model we introduced a new approach or possibilities for the development of BCIs.
New approaches to brain- computer interfaces (BCI)
 We show that in spite of the fact that detected brain signals represent a superposition of all the active neurons,
it is still possible to decode some information that is transferred between the active regions of the brain when
the cooperation of the regions is necessary to accomplish the task.
 BCI development - considering the computational time to pre-process the data, a real-time online analysis
would be possible if the real-time non-causal limitations of the methodology could be overcome. But non-
determinicity of brain is a limitation.
 Neuroprosthetics
1. devices that can substitute a motor, sensory or cognitive modality that might have been damaged as a
result of an injury or a disease
2. implantable devices can also be used in animal experiments as a tool for neuroscientists in order to de-
velop a better understanding of how the brain works. Wireless neuro recording from the brain of awake,
freely behaving animals can open many important doors into understanding how the brain handles different
functions.
MAIN REFERENCES
 Using ANNs to predict a subject’s response based on EEG traces. Logar V, Belic A, Koritnik B, Brezan S, Zidar J, Karba R, Matko D. Neural
Netw. 2008 Sep;21(7):881-7.
 Gripping-force identification using EEG and phase-demodulation approach. Logar V, Skrjanc I, Belic A, Karba R, Brezan S, Koritnik B,
Zidar J. Neurosci Res. 2008 Apr;60(4):389-96.
 Logar V, Skrjanc I, Belic A, Brezan S, Koritnik B, Zidar J. Identification of the phase code in an EEG during gripping-force tasks: a possible
alternative approach to the development of the brain-computer interfaces. Artif Intell Med.2008 Sep;44(1):41-9
 Baddeley, A. (2000). The episodic buffer: a new component of working memory? Trends in cognitive science, 4(11), 417-423.
 Schnitzler, A., Gross, J., 2005. Normal and pathological oscillatory communication n the brain. Nat. Rev./Neurosci. 6, 285–296.
 Singer, W., Gray, C.M., 1995. Visual feature integration and the temporal correlation hypothesis. Annu. Rev. Neurosci. 18, 555–586.
 Buzsaki, G., Draguhn, A., 2004. Neuronal oscillations in cortical networks. Science 304 (5679), 1926–1929.
 Fuster J (2000) Cortical dynamics of memory. International Journal of Psychophysiology 35:155-164
 Engel, A.K., Fries, P., Singer,W., 2001. Dynamic predictions: oscillations and synchrony in top-down processing. Nat. Rev. Neurosci. 2 (10),
704–716.
 Basar E, Basar-Eroglu C, Karakas S, Schurmann M. Gamma, alpha, delta, and theta oscillations govern cognitive processes. Int J Psycho-
physiol. 2001 Jan;39(2-3):241-8
 Jensen, O ., Lisman, J.E. (1998). An oscillatory short-term memory buffer model can account for data on the sternberg task. The journal of
neuroscience, 18(24), 10688-10699.
 Klimesch W. (1996). Memory processes, brain oscillations and EEG synchronization. Internal journal of psychophysiology, 24, 6-100
 Von Stein, A., & Sarnthein, J. (2000). Different frequencies for different scales of cortical integration: From local gamma to long range al-
pha/theta synchronization. International Journal of Psychophysiology, 38(3), 301–313.
 Jensen O and Lisman J E (2005) Hippocampal sequence-encoding driven by a cortical multi-item working memory buffer. Trends in Neu-
rosciences 28(2):67-72
 Kahana M J, Seelig D, Madsen J R (2001) Theta returns. Current Opinion in Neurobiology 11:739-744
 Sternberg S (1966) High-speed scanning in human memory. Science 153:652-654