Summary 5

BSE662A: Decision Making and The Brain

The Brain’s Habit Machinery

The human brain is made up of tens of billions of neurons (primary cells that
process information) and a large number of glia (support cells). Before sending
out chemical signals that alter the electrical activity of neighbouring neurons,
neurons transfer electrical impulses from one end to the other. This confluence of
electrical and chemical signals, as well as the structure of our brains, which
dictates which neurons are connected to which ones, govern everything we do.
Coco's brain is one illustration of this when she notices a bird outside the window.
The light reflected by the bird is taken in by the retina of Coco's eye, which has
specialised neurons that can perceive light. A "action potential" is the name given
to this electrical signal, which travels the entire length of the cell. Chemicals begin
to leak out as a result of this signal, and the subsequent cell in the chain may
then be turned on or off. These messages go through multiple layers of neurons
in the eye before passing through a nerve, which is a wire made up of numerous
axons, to eventually reach the brain. The thalamus, a deep-brain region that can
be compared to the brain, is where the information first enters the body. The
thalamus, which can be considered to be the brain, is where the signal first enters
the brain. Different regions of the cerebral cortex receive different types of
information, such as information from the eyes. Individual neurons are only
receptive to information from tiny regions of the visual environment and to simple
features in regions of the visual cortex that receive signals from the thalamus.
These cues eventually lead Coco to identify the pattern in the visual input as a
bird. This causes the release of neurochemicals, which in turn transmits messages
to other regions of the brain that regulate emotions. These agitate and thrill her
much. Some of these messages also activate her motor centre, causing her to

1
move quickly toward the door while making odd noises.

It's critical to comprehend how the brain functions in order to comprehend how
habits develop and how long they stay. When you're doing them and later on
when you think about them, habits might persist even though you have no idea
where they came from. This is particularly evident in individuals who have lost
their memories as a result of brain injuries. Édouard Claparède, a French doctor,
is a well-known example of this. One of his patients remembers being poked with
a pin that was tucked away in his palm, he discovered. There are several memory
systems in the brain, according to neuroscientific research that dates back to the
1960s. The primary distinction lies between memory systems that enable
conscious recollection and those that don't. Conscious memory is a requirement
for some memory systems, while it is not a requirement for others. The ability to
consciously recall the past is controlled by the declarative memory system. It is
made up of a collection of brain regions located in the middle temporal lobe, a
deep region of the temporal lobe. It may be impossible to remember the past or
create new memories if these areas of the brain are damaged. By examining the
brain of a guy with the initials R. B. who experienced severe memory issues after
experiencing cardiac episodes that temporarily deprived his brain of oxygen, Larry
Squire and his colleagues were able to demonstrate this. His intelligence was
unaffected, but his memory for new information was severely compromised. After
having surgery to manage his severe epilepsy, Henry Molaison lost his memory.
Henry Molaison was investigated by Brenda Milner and Suzanne Corkin after he
underwent surgery to treat his severe epilepsy and became amnesic. They
discovered that while he was unable to remember the past consciously, he was
able to learn and maintain a number of motor abilities rather effectively.
Both Parkinson's disease and Huntington's disease are characterised by the
degradation of dopamine neurons in the substantia nigra, which causes a relative
rise in the indirect pathway's activity and a decrease in the direct pathway's
activity. Dopamine has an impact on synaptic plasticity, the basic process of
2
change in the brain. Excitatory neurotransmitters (glutamate), which attach to the
striatal neuron, must be released in order for this process to take place. These
chemicals are released into the region of the striatum that lies between the axon
and its target neuron. Some of them come into contact with receptors that are
located on the neuron's surface across the synapse. These two illnesses highlight
how crucial it is for activity to be distributed evenly across these two pathways for
the brain to function properly. The process through which experience modifies
the strength of synapses, causing some neurons to become more effective at
activating other neurons while others become less effective, is known as synaptic
plasticity. Although dopamine is not a direct cause of neuronal plasticity, it is a
crucial component of the three-factor rule that determines how plasticity is
altered. One of the most prevalent types of flexibility is when one cell prompts
another to fire immediately after it, strengthening the bond between them.
Dopamine and rewards have long been associated. This is because prior research
demonstrated that rats will engage in virtually any behaviour to receive
stimulation from an electrode placed in a region of the brain that causes the
release of dopamine. The early research Schultz did with monkeys demonstrated
that positive experiences actually activate the brain's dopamine neurons. But he
observed something that might alter the way we see dopamine's functions. When
the monkey unexpectedly received a treat, the dopamine neurons would start to
fire. Dopamine neurons started to fire only when the light turned on after the
monkey realised that the light signalled the reward was approaching. This
discovery was crucial because it connected dopamine to a group of computer
science and psychology theories that gave rise to the approach that is currently
the most well-liked for using computers to understand what dopamine
accomplishes.

Making systems that can learn from experience, like machine learning, which
involves learning by trying things out and seeing what works and what doesn't,
has long piqued interest in the field of computer science. The concept of
3
reinforcement learning dictates what one should do in situations like this.
According to the principle of reinforcement learning, we should gain knowledge
depending on how well our assumptions correspond to reality. According to the
majority of theories on reinforcement learning, the decision-maker chooses an
action based on how valuable they believe it will be in comparison to all the other
actions they may do. They then modify their estimate for the following round in
light of this information.

Researcher Ilana Witten works at Princeton University. To understand how

dopamine functions in all of its many nuances, she wants to research mice.
Working with David Tank, a different Princeton professor whose team had
developed a virtual reality system for mice, she was able to do this. The mouse in
this system is perched on a little ball that resembles a ping-pong ball. It runs on
the ball while watching a realistic video show with a little metal mask holding its
head in place. In 2019, researchers recorded from neurons in an animal's brain as
it was acting using a technique called "calcium imaging." They were able to
determine how the activity of neurons in the dopamine system is related to more
than just errors in reward prediction using this strategy.
More than 300 dopamine neurons in the brains of 20 mice were monitored by
Witten and her colleagues as the mice engaged in a game in which they ran
down a virtual hallway and then made a left or right turn. Dopamine neurons
would only fire when a reward arrived unexpectedly or when a sign indicated that
a reward was on the way if the reward prediction error theory were accurate.
Witten's team used a statistical model created by Nathaniel Daw, another
Princeton professor, to examine how dopamine neurons responded to a variety of
mouse actions, including where the mouse was in the corridor, how quickly it was
moving, and if it had received a reward on the previous session. Dopamine
neurons are shown in Witten's study to respond to various game-related
elements, but this isn't the only factor that causes them to do so. According to a
study by John Salamone, rats are more likely to select the lower amount of food
4
that doesn't require them to work for it if dopamine levels are altered. This is not
due to the animals' inability to scale the barrier due to dopamine disorders. They
will climb the barrier to obtain the food if forced to choose between doing so and
going without, but it appears to reduce their willingness to labour for food.
Salamone's findings are consistent with those of neuroscientists Kent Berridge
and Terry Robinson, who claim that dopamine is what they refer to as "incentive
salience"—a signal that indicates how eagerly an organism wants something and
how hard it will work to obtain it.

Although their function is unclear, the dopamine receptors in the nucleus

accumbens appear to constitute an important component of drive. It doesn't
appear that dopamine blockade in the nucleus accumbens alters the animal's
fundamental appetite or the pleasure it experiences when consuming food. The
animal is less inclined to go through the necessary steps to obtain food or to work
harder in order to obtain more food, nevertheless. Cannabis and opioid chemical
systems in the brain are two further examples of how gratifying something is.
These findings are supported by studies on naltrexone, a medication that blocks
the effects of opioids and is frequently used to treat intoxication. According to
studies, naltrexone lessens the enjoyment of a wide range of behaviours,
including sex, gambling, and amphetamine use.

The computations that aid in helping us choose what to do and how to do it at

any given time appear to be centred on the basal ganglia and the dopamine
system. According to Peter Redgrave, the basal ganglia function as a sort of
"central switchboard" for decision-making. The basal ganglia receive information
from the brain that represents potential actions. The globus pallidus possesses
significant tonic inhibition, which prevents any movements from occurring,
supporting this theory. The computations that aid in helping us choose what to do
and how to do it at any given time appear to be centred on the basal ganglia and
5
the dopamine system. The study of Rui Costa at Columbia University has very
clearly demonstrated how action selection functions in the basal ganglia. To
determine which pathway each neuron in the striatum was on, he employed
technologies from optogenetics. Then, for each pathway, he examined the
activity of those neurons separately. They trained mice to press levers in a
specific order as part of their experiment. They discovered that at the beginning
of the process, both direct and indirect pathway neurons became active, but only
direct pathway neurons remained active at the conclusion. According to this
study, the basal ganglia are where the brain forms new habits, and dopamine
plays a crucial role in this process.

Making long-lasting behavioural changes requires a thorough understanding of

the factors that underlie habit formation. It can be challenging to break habits
because they can be firmly embedded in our minds and work on a subconscious
level. Two buttons are presented to a rat. The rat receives a bit of food from one
of the levers. One of the two button presses will result in the winner of the race
being determined by the brain's instructions to the striatum. Dopamine is released
if the rat eats. This strengthens the connections between the brain and the
striatum, increasing the likelihood that the cerebral cell will trigger striatal neuron
firing the following time. The amount of dopamine in the rat's brain decreases if it
is denied food. The likelihood that the rat will select the action that provided food
the next time it had the same choice is increased as a result. The rat is then more
likely to select the awarded action the following time it has the same choice as a
result of these changes in the strength of the links. This eventually turns into a
habit.

6
Once a Habit, Always a Habit
We need to understand why habits are so difficult to break in order to comprehend
how someone could quit using for a very long period before resuming their use.
When to remain fixed and when to change is a decision the brain must make. The
stability-plasticity dilemma is what's known as this. For the past 20 years, Mark
Bouton from the University of Vermont has been attempting to understand why it is
so difficult to stop bad behaviours. The phenomena that Bouton has examined go by
several names, but they all seem to be brought on by the same phenomenon: habits
that appeared to have vanished suddenly reappear. When we replace an old habit
with a new one, we don't forget the previous habit, according to Bouton's study on
resurrection and related topics. Instead, we consciously discontinue the old
behaviour to make room for the new one. For many of the harmful habits we want
to break, this is especially crucial.

UCLA psychologist Michelle Craske has researched methods to manage anxiety

disorders. She and her coworkers published a study in 2002 detailing how they
assisted a group of college students who were terrified of spiders. The Chilean Rose-
Haired Spider (Phrixotrichus spatulata, with legs spread of around 6 inches or 15.2
cm) served as the experiment's phobic trigger. The pupils moved from standing close
to the spider and touching it with a covered hand to having the spider walk on their
naked hand in the space of an hour. By the end of the lesson, the majority of the
students were able to let the spider walk on their palm thanks to how well the
treatment had worked.
Since the students having the follow-up in the same room as the initial treatment
showed less fear than those who saw the spider again in a new setting, Craske and
her colleagues came to the conclusion that changing the context diminished the
therapy's effectiveness. The habit system avoids the stability vs. flexibility issue by
presuming that nothing ever changes. This implies that whatever habit forms first (in

7
response to a specific incident) is taken to represent how the world remains
unchanged. This has repercussions for how we might approach kicking bad
behaviours. A common unhealthy habit that can harm one's fingernails and lead to
dental issues is biting one's fingernails. Henry Yin, Barbara Knowlton, and Bernard
Balleine from UCLA conducted a number of investigations that were published in
2004 and 2005 to determine this.
These investigations demonstrated that various basal ganglia regions play various
roles in habit formation. A model developed by Yin and Knowlton explains how the
brain develops habits. The paradigm begins with goal-directed learning, which makes
use of the "cognitive" corticostriatal loop connecting the caudate nucleus and
dorsolateral prefrontal cortex. The "motor" circuit, which is composed of the
putamen and motor cortex, begins to learn the pattern over time and finally replaces
the "cognitive" loop.

When an action learnt through instrumental learning is triggered by a cue learned

through Pavlovian learning (such as the sound of someone else's phone ringing), this
is known as a "Pavlovian-instrumental transfer" (like checking for messages). Since it
might induce the need to immediately fire up a cigarette in a smoker, it is believed to
be crucial in the development of unhealthy habits. The rats are more likely to press
the lever when the sound is playing than when a different sound that isn't connected
with food is played, according to studies with rats, demonstrating that Pavlovian-
instrumental transfer is taking place. Since it may prompt a smoker to grab a
cigarette right quickly, this is regarded to be particularly crucial when developing
harmful habits.
Holland discovered that long-trained rats on the instrumental lever press exhibited
high Pavlovian-instrumental transfer, a hallmark of routine behaviour. The impact of
prize reduction on Pavlovian instrumental transfer was then examined. Rats with
minimal training demonstrated little transfer, but rats with lots of training displayed
lots of transfer that persisted even after the food's value decreased. According to
research, people do exhibit Pavlovian-instrumental transfer, and habits are
8
particularly susceptible to it. Sanne de Wit and her colleagues conducted a study in
which five participants were instructed to press one of two buttons to get either
popcorn or chocolate chocolates. They discovered that hearing a sound connected
with food increased dopamine levels in the rat's nucleus accumbens compared to
hearing a sound unrelated to food. When they examined the timing of dopamine
release, they discovered that the rats frequently released more dopamine just prior
to pulling the lever. This demonstrates that dopamine and Pavlovian-instrumental
transfer are more closely related.
The study's findings were suggestive but only correlative; to establish a causal
relationship between dopamine and Pavlovian-instrumental transfer, it is necessary
to demonstrate that affecting dopamine also affects transfer. A study by Sean
Ostlund, Kate Wassum, and others demonstrated this. To block dopamine impulses
in particular regions of the brain, they used a novel technique known as DREADDs.
The nucleus accumbens dopamine signal blocking decreased the Pavlovian-
instrumental transfer. However, the transfer was unaffected when dopamine
impulses in a different area of the brain (the rat's medial prefrontal cortex) were
blocked. This demonstrates that dopamine in the nucleus accumbens plays a
significant role in the relationship between reward-seeking behaviours and cue
triggers.

Individuals with drug addiction frequently experience attentional bias. These folks
have a propensity to pay close attention to visual clues related to their preferred
medication. This has been demonstrated in a Stroop task variation where drug users
are shown images or words associated with their addiction and instructed to respond
to a simple stimulus, such as the colour of an image. People who utilise drugs react
to events more slowly than those who do not. This implies that material about drugs
automatically catches their attention. Brian Anderson, a psychologist at Texas A&M
University who has studied this effect, contends that the attentional biases seen in
addiction are not abnormal but rather a representation of a fundamental
psychological process called value-based attentional capture. Anderson discovered
9
that highly rewarded colours were more perplexing than poorly rewarded colours,
and that people took longer to find the goal form when there was a distractor in
either of the awarded colours. According to Anderson, value-based attentional
capture is ingrained and challenging to overcome, similar to the attentional biases
present in addiction. Additionally, he discovered that the components of the
movement come together to form a single unit (or "chunk") that, once begun,
proceeds naturally to completion, making it much simpler to begin the practise.

Because of the mechanisms Mark Bouton's research has identified, habits are
challenging to break. A goal-directed action, like as checking texts on a smartphone,
gradually shifts from relying on corticostriatal loops to relying on motor function. This
implies that the cognitive system is no longer required to directly monitor it. The
likelihood is that things will resume being done the old way if the situation changes.
When someone's phone buzzes with a new message, for example, mechanisms like
Pavlovian-instrumental transfer and value-based attentional capture combine to
make it simpler to activate the habit. Our behaviours are a result of both our habits
and the things we undertake to accomplish our objectives.

10