UCSB RMP

Q-Learning for Deep Brain Stimulation AUG 2019

!

Utilizing Q-Learning to Optimize Deep Brain Stimulation in Parkinson’s Patients

Rahul Kumar,1 Timothy Matchen2, Jeff Moehlis2
1Indus
International School, Billapura Cross, Sarjapura, Bangalore, Karnataka
rahulukumar02@gmail.com
2Department of Mechanical Engineering, University of California, Santa Barbara, CA 93106


Over 6 million people around the world are affected by Parkinson’s Disease, a degenerative disorder that affects the
motor system. The loss of dopaminergic cells in the striatum is normally treated with levodopa, a precursor to
dopamine which acts a dopamine replacement agent. However, levodopa’s efficacy reduces with time, as well as
causing adverse side effects such as dyskinesia, and so levodopa is not viable as a long-term cure. Deep Brain
Stimulation (DBS) is a surgical method designed to reduce the effects of Parkinson’s by placing a neurostimulator in
the brain which sends electrical impulses through electrodes in order to treat movement disorders such as
Parkinson’s. However, DBS is not well-understood, and optimizing stimulation parameters is done manually, which
is a time-consuming and difficult task. If the stimulation parameters could be found electronically rather than
manually, time and effort could be saved. Furthermore, by ensuring that the parameters are as optimal as possible,
the patient would get the greatest benefit. Since Parkinsonian neurons show excess synchronization, it is necessary
to desynchronize the neurons via a stimulator. Current iterations of neural networks control this process by offsetting
the firing of each stimulator. However, it is not necessary that the offsets used are the most efficient. We propose a
Q-Learning solution in which the offsets are not predetermined, in order to find the optimal stimulation parameters
as quickly as possible.

Keywords: Parkinson’s Disease, Deep Brain Stimulation, Machine Learning, Reinforcement Learning, Q-Learning

1. INTRODUCTION order for DBS to be as effective as possible, several

Parkinson’s Disease (PD) is a degenerative disorder of
the central nervous system for which there is no known
cure. Over 6 million people worldwide suffer from PD, and
it causes over 100,000 deaths each year [1]. PD sufferers
experience several motor symptoms such as shaking,
rigidity, and bradykinesia, caused by the death of cells in
the substantial nigra, resulting in a dopamine deficiency [2].
As PD progresses, patients often demonstrate non-motor
symptoms such as dementia, depression, sleep deprivation,
and behavioral issues [3].
While PD cannot be cured, the symptoms can be partially
alleviated by levodopa, or L-DOPA [3]. The dopamine
molecule itself is too polar to pass through the blood-brain
barrier, but L-DOPA, as an amino acid, can [4]. In the
brain, L-DOPA is converted into dopamine via the process
of decarboxylation. However, since PD is a degenerative
disease, L-DOPA becomes less and less effective as the
disorder progresses [5].
When PD can no longer be controlled through medication
alone, Deep Brain Stimulation (DBS) is used to control the
side effects, especially motor fluctuations and tremors [6].
DBS is the electrical stimulation of areas of the brain,
including the globus pallidus internus, thalamus,
subthalamic nucleus, and the peduncolopontine nucleus by
a neurostimulator. Each patient has different symptoms, and
so the specific location of DBS in the aforementioned areas
differs. DBS is highly effective, and patients show
significant improvement in motor score evaluations [7]. In
parameters, such as frequency, amplitude, and pulse width
must be optimized [8]. This optimization, which is done
manually, is time-consuming for both the surgeon as well as
the PD patient.
Therefore, we wish to automate this process by adjusting
the above process based off of electrical feedback from the
patient. Machine learning models for deep brain stimulation
have been made before, utilizing closed-loop models, deep
learning classification, and optimization algorithms.
However, these models are not optimized, and therefore do
not approach the efficiency needed for DBS.
In this paper, we propose a solution using a
reinforcement learning neural network which utilizes Q-
Learning. Reinforcement learning algorithms do not require
input/output pairs, or explicit correction. Instead, they strike
a balance between using current knowledge and obtaining
new knowledge to find an optimal solution.
Unlike supervised learning, reinforcement learning does
not require a target output. Instead, the agent is given
feedback about the appropriateness of its response.
However, the agent is not told what the correct response
should be, unlike in supervised learning. For this reason,
reinforcement learning is much more realistic than
supervised learning - in real life and in nature, the agent
rarely knows the right answer, only that it has made a
mistake.
 UCSB RMP
Q-Learning for Deep Brain Stimulation
!
II. METHODS
Patients with Parkinson’s Disease exhibit excessively
synchronized neurons in the brain. Due to this, the brain
struggles to control the motion of the body, leading to the
motor symptoms commonly experienced in PD such as
tremors, shaking, and bradykinesia. Deep Brain Stimulation
desynchronizes the neurons, which helps the patient
function normally. Our goal is to use Q-Learning to
automatically stimulate the patient. In order to model the
activity of neurons in the brain, we used a network of phase
oscillators, which represent our Parkinsonian neurons. The
activity of these oscillators is modeled by Equation 1 in
Tass 2003[9].The oscillators, like the neurons they
represent have a tendency to clump together. For this
reason, before we turn on our stimulators, our oscillators
are all at the same phase. Our model’s aim is to use our
stimulators to spread apart the phase of the oscillators. We
use Equation 4 in Tass 2003[9] to measure the level of
synchronization in the neurons.
We built a Q-Learning model which decides whether or
not to turn the neurostimulator on. The environment of the
model consists of neurons, which are assigned random
eigenfrequencies. In order to closely model patients with
Parkinson’s Disease, the neurons are closely synchronized -
their eigenfrequencies are similar. We assign a learning rate,
discount factor, and exploration/exploitation rate to the
model.
The Q-table has two states: Phase2 and R2. There are
only two actions: either the neurostimulator is turned on, or
it is turned off. The reward is calculated by measuring the
synchronization of the neurons - the further apart the
eigenfrequencies, the higher the reward. The goal of our Q-
Learning model is to maximize the reward by
desynchronizing the neurons.
The model starts out by exploring the state and action
space. Since the learning rate is originally very high, the
agent almost always pick a random action. As the agent
learns more, the learning rate decreases. The agent begins
to use its own judgement to maximize its reward.
The agent’s discount factor determines the weight it
places on current rewards vs future rewards. If the discount
factor is too high(the agent cares too much about current
rewards), the agent is less likely to converge to the highest
reward. For this reason, the agent starts with a low discount
factor which gradually increases over time.
The first iteration of our model had only one stimulator,
and so all of the neurons were placed in the same group:
our stimulator had knowledge of all of their
eigenfrequencies. Our Q-learning agent successfully
desynchronized the neurons.
However, stimulating each neuron individually would
require many electrodes[9], so we decided to split our
neurons into multiple populations. Each population would
be controlled by a different stimulator. These stimulators
only have information about the set of neurons they control:
they are not aware of the frequencies of other neurons.
Their reward is calculated by only measuring the
synchronization of neurons that the stimulator controls.
AUG 2019
Originally, we used 4 stimulators, as this was the
accepted number in previous literature[9]. We also
experimented with different numbers of stimulators to find
the optimum.
III. RESULTS
Our first and main goal was to build a working Q-
learning algorithm that could desynchronize our population
of neurons. Our algorithm largely succeeded in that task.
Our neurons started off completely synchronized, and
ended up largely desynchronized(See Appendix 1).
Completely desynchronizing our population is impossible,
as the neurons are globally coupled and thus resist
desynchronization. However, our algorithm could be
optimized to get the extent of desynchronization higher.
Our second goal was to test if we could achieve a similar
level of desynchronization with multiple populations of
neurons, all hidden from each other. In other words, each
population was completely separate, with no interaction or
knowledge flow between different stimulators. Firstly, we
found that our model with separate populations did an
effective job of desynchronizing our neurons.(See
Appendix 2) This is important, as it is impractical to
stimulate each neuron separately. By dividing the neurons
into groups and stimulating each group as a whole, we
reduce the amount of electricity applied to the neural tissue,
and thus reduce the chance of damage.
A previous study on this subject succeeded in
desynchronizing neurons by separating them into separate
groups. This study demonstrated that the optimal
stimulation pattern for multiple stimulators was to have
them offset from each other: each would peak at different
times. Therefore, this behavior was hard coded into the
stimulators. On the other hand, while our stimulators were
given no such information, they learned the same type of
behavior by themselves.(See Appendix 3) Over time, the
stimulators begin to fire offset from each other. This tell us
that it is not necessary to hard code that behavior into our
stimulators, saving time.
Next, we wanted to find if 4 stimulators was indeed the
optimum. For this reason, we ran our model again, with the
number of stimulators varying from 3 to 8. We found that,
in our model, the optimum number of stimulators was
indeed 4. This is because the model runs into issues if the
number of stimulators is either increased or decreased.
If the number of stimulators is increased, each stimulator
controls an increasingly smaller slice of the population. For
example, if there are 8 stimulators, each stimulator only
controls 12.5% of the environment, and has no information
about the remaining 87.5%. For this reason, the stimulators
cannot desynchronize the entire population, as they do not
know the frequency values of most of the neurons. In many
runs of our model, the neurons tend to clump together as
they are globally coupled and the stimulators do not take
explicit action to prevent this.
With a smaller number of stimulators, we faced the
opposite problem. As mentioned above, the neurons want to
 UCSB RMP
Q-Learning for Deep Brain Stimulation
!
synchronize, and must be forced apart by the stimulators.
Neurons in the same population usually end up with similar
frequencies, as the stimulator always performs the same
action on its population. It cannot choose to stimulate only
half of its population; it must stimulate all or none. On the
other hand, different stimulators perform different actions,
and so different populations of neurons diverge rapidly.
With a smaller amount of stimulators(2 or 3), the
population is not pulled apart as effectively.
IV. CONCLUSIONS
Q-Learning can be used effectively in Deep Brain
Stimulation. Our model successfully desynchronizes the
neuronal population. However, our original model is not
viable as stimulating each neuron would involve pumping
far too much electricity into the patient.
Our second model is far more viable as it significantly
limits the number of stimulators, and thus the danger of
damage to the patient. We find that 4 stimulators do almost
as well as stimulating each neuron individually.
We also find that it is unnecessary to force the stimulators
to fire at different times. While that is the most efficient
firing pattern, our stimulators learn to do so by themselves,
therefore saving the operator’s time.
Finally, 4 stimulators, the accepted amount in previous
studies, is indeed the optimal for our setup. Increasing or
decreasing the number of stimulators decreases the extent
of the desynchronization.
V. FUTURE WORK
There are several ways in which we can expand the work
that we have done so far. Currently, our stimulators receive
no knowledge about the actions of nearby stimulators. This
is the reason why their effect drops significantly as the
number of stimulators increases. It is possible that a larger
amount of stimulators would be more effective if each
stimulator received partial information - the actions of its
immediate neighbors. We would like to see if the extent of
desynchronization increases with this change.
Secondly, our Q-Learning algorithm has numerous
dependent variables, such as learning rate, exploitation-
exploration coefficient, discount factor, intensity, strength
of inter-neuronal attraction, as well as number of
stimulators. We would like to vary these variables in order
to see if there is any correlation between them and
desynchronization of the neurons.
Furthermore, in some tests of our model, some
stimulators seem to abruptly turn off, though this does not
affect the extent of desynchronization. Since Parkinsonian
neurons tend to fire at similar frequencies, it is odd that the
neurons do not tend towards synchronization once the
stimulators stop acting on them. Further research is
required to understand if this is simply an error in our
model or if the neurons truly do manage to stay
desynchronized.
AUG 2019
ACKNOWLEDGEMENTS
The first author would like to thank his co-authors for
their continued mentorship and guidance throughout this
project. Their insight and technical ability have been
invaluable, and their scientific ability was crucial to this
project. We would like to thank the rest of the Moehlis lab
for their assistance and support. We would also like to
thank the University of California Santa Barbara for
providing us with the facilities we required in order to
complete this project, as well as Dr. Lina Kim, Dr. Michael
Hughes, and the rest of the Research Mentorship Program
for giving us the ability to undertake and complete this
project.
REFERENCES
[1] Sejvar, James. “Faculty of 1000 Evaluation for Global,
Regional, and National Burden of Neurological
Disorders during 1990-2015: a Systematic
Analysis for the Global Burden of Disease
Study 2015.” F1000 - Post-Publication Peer
Review of the Biomedical Literature, 2018
[2] “Parkinson's Disease Information Page.” National
Institute of Neurological Disorders and Stroke, U.S.
Department of Health and Human Services,
www.ninds.nih.gov/Disorders/All-Disorders/
Parkinsons-Disease-Information-Page.
[3] Sveinbjornsdottir, Sigurlaug. “The Clinical Symptoms
of Parkinson's Disease.” Journal of Neurochemistry,
vol. 139, 2016, pp. 318–324.
[4] “L-DOPA.” L-DOPA - an Overview | ScienceDirect
Topics, www.sciencedirect.com/topics/neuroscience/l-
dopa.
[5] Thanvi, B R. “Long Term Motor Complications of
Levodopa: Clinical Features, Mechanisms, and
Management Strategies.” Postgraduate Medical
Journal, vol. 80, no. 946, 2004, pp. 452–458.
[6] Montgomery, Erwin B. “What Is Deep Brain
Stimulation?” 20 Things to Know about Deep Brain
Stimulation, 2014, pp. 1–13.
[7] “Deep Brain Stimulation.” AANS, www.aans.org/en/
Patients/Neurosurgical-Conditions-and-Treatments/
Deep-Brain-Stimulation.
[8] Mcintyre, Cameron C., et al. “Optimizing Deep Brain
Stimulation Parameter Selection with Detailed Models
of the Electrode-Tissue Interface.” 2006 International
Conference of the IEEE Engineering in Medicine and
Biology Society, 2006
[9] Tass, Peter A. “A Model of Desynchronizing Deep
Brain Stimulation with a Demand-Controlled
Coordinated Reset of Neural Subpopulations.”
Biological Cybernetics, vol. 89, no. 2, 2003, pp. 81–88.
 UCSB RMP
Q-Learning for Deep Brain Stimulation AUG 2019
!

APPENDIX FIG. 2. (Color online) A graphical representation of Fig. 1.

The initial part of the graph shows the synchronization of
phase oscillators without DBS. The right-hand side of the
1. Desynchronization graph shows the rapid desynchronization of the oscillators
once stimulation is applied.
a. Extent of desynchronization

2. Stimulator offset

FIG. 1. (Color online) The extent of desynchronization,

measured by Equation 4 in Tass 2003[9]. Our model starts
with a set of synchronized phase oscillators and
desynchronizes them effectively. FIG. 3. (Color online) Each line represents a different
stimulator. While the stimulators initially fire at random
without any pattern, they eventually learn to peak offset
b. Graphical representation
from the other stimulators - they all peak at different times.

3. Different number of stimulators

a. 3 Stimulators
 UCSB RMP
Q-Learning for Deep Brain Stimulation AUG 2019
!

FIG. 4. (Color online) With 3 stimulators, the different

stimulators show no pattern in their firing, likely because
there are too few stimulators to force the oscillators apart

b. 6 Stimulators

FIG. 5. (Color online) Our model with 6 stimulators shows

the same characteristic offset. However, it does not do as
well, likely because the stimulators do not have enough
information. In this graph, we notice that some stimulators
occasionally “turn off”.