Mind Reading Computers

Animesh Gupta, Ankit Anand, Rubina Arora

Manipal Institute of Technology, Manipal
animesh_15iway@yahoo.com

Abstract--The ease of interaction with a computer system II. SPECIFIC USES

depends primarily on how good the interface is between the
user and the system. The most efficient form of interaction
is when a computer can perceive the emotions and thought
process of its user by reading their facial expressions. This
technology can enhance living of paralytic people or people
who can’t speak by providing them with an interaction
platform. Other areas where it can be implemented are
military sting operations, robotics, gaming consoles amongst
many others. The mind reading actually involves measuring
the volume and oxygen level of the blood around the
subject’s brain using a technology called functional
near-infrared spectroscopy (fNIRS). This paper deals with
the design and development of mind reading computers and
gives a brief overview of the factors involved for its further
enhancement.
Keywords— Computer, Human, Interaction, Robotics, Medicine
I. INTRODUCTION
A mind reading machine is basically co-ordination between
human psychology and computer techniques. Knowledge of
how particular mental states are expressed in the face is
combined with analysis of facial expressions and head
gestures occurring in real time. Using a digital video camera,
the mind-reading computer system analyses a person's facial
expressions in real time and infers that person's underlying
mental state, such as whether he or she is agreeing or
disagreeing, interested or bored, thinking or confused. As an
example, a combination of head node, with a smile and
eyebrows raised might mean that a person is interested.
There are various software designed for face recognition. One
such software from Nevenvision identifies 24 feature points
on the face and tracks them in real time. Movement, shape and
colour are then analysed to identify gestures like a smile or
eyebrows being raised. Combinations of these occurring over
time indicate mental states.
The most important thing which we have realized through the
course of research for this paper is ‘It’s nearly impossible to
bluff this software’ since it’s very difficult for one to express
anger with a happy face.
Imagine a future where we are surrounded with mobile
phones, cars and online services that can read our minds and
react to our moods. Such a technology could revolutionise the
interaction scenario between computers and humans.
MEDICAL RELEVANCE
The most noteworthy use of a mind reading system is
in communication with people in coma. Lately, these
systems have proven to be efficient in deducing
answers to specific ‘yes’ or ‘no’ questions. The
technology uses neuroimaging to read human
thoughts via brain activity to detect ‘yes’ or ‘no’. It
only requires the participants to concentrate on
his/her response.
Considering questions like ‘Are you feeling better?’
or ‘Do you have any kids?’ one can easily
communicate with a person (rather patient) in an
unconscious or paralytic state. This could lead to
dramatic new ways of attempting to communicate
with patients in a vegetative state.
The technology can be used in understanding the
thoughts of patients who are fully conscious and
awake, yet, due to brain damage, are unable to show
any behavioral responsivity. The interpretation of
human thoughts from brain activity, without
depending on speech or action is one of the most
challenging areas of modern neuroscience.
Fragments of people's thoughts are picked by
decoding the brain activity caused by words that they
hear. Such devices could transform the lives of
thousands of people who lose the ability to speak as a
result of a stroke or other medical conditions. The
prospect of devices that can return speech to the
speechless is huge.

Figure1. Mind Reading headgear

 Curbing criminal activity
Mind reading systems can serve as a powerful tool
that allows anyone to look deep inside a person's
brain and read their intentions before they act. The
interrogators can eavesdrop on the thoughts of the
criminals and thus it also raises serious ethical issues
over how brain-reading technology may be used in
the future. The system makes use of high-resolution
brain scans to identify patterns of activity before
translating them into meaningful thoughts, revealing
what a person planned to do in the near future. It is
the first time scientists have succeeded in reading
intentions in this way.
Figure1. Brain Computer Interface (BCI)
III. GENERAL USES
• E-learning
Apart from uses in neuroscience, a mind reading computer can
In e-learning applications, affective computing can
be implemented in our day to day lives as well. Some of the
major areas where such a system can successfully be be used to adjust the presentation style of a
implemented are: computerized tutor when a learner is bored,
• Detecting driver mental states interested, frustrated, or pleased which would act as
A large number of accidents occur every year due to instant and honest feedback.
sleep deprivation in the drivers. A mind reading • Robotic systems
computer system can be installed in cars to detect the The best type of robotic helper is one who can read
mental state of the driver such as drowsiness, your mind. The user sends a command to the robot to
distraction and anger. The system can read and process into a movement. This requires programming
process the input and give appropriate commands to the robot with a predefined set of very basic
the hardware. This can tremendously reduce the behaviors. Scientists are working on creating robotic
possibility of an accident. wheelchairs that can be manipulated by thought.
• On-Line Shopping
We are also looking at the use of mind-reading to IV. WORKING
support on-line shopping and learning systems. For
There are various steps involved in the complete
example, while going through a collection of shirts, implementation of a mind reading computer. The steps can
the user can express interest by a wink or any other broadly be categorised as:
appropriate gesture and the mind reading computer • Reading the facial expressions or gestures of the
can re-direct to the payment page based on that person
gesture. • Processing the input by running it through an in-built
• Human computer interaction algorithm which analyses the commands.
Affective computing has potential applications in • Displaying the output to the user or performing the
human computer interaction, such as affective desired task.
mirrors allowing the user to see how he or she
performs, emotion monitoring agents sending a
warning before one sends an angry email or even
music players selecting tracks based on mood.

Figure2. Brain scanning by a mind reading computer

It all begins with passive sensors which capture data about the
user's physical state or behavior. The data gathered is similar
to the cues which humans use to perceive emotions in others.
A video camera captures facial expressions, body posture and
gestures while a microphone captures speech. There are
various other sensors attached to a mind-reading computer
which measure psychological data such as skin temperature
and galvanic resistance.
After the capturing and collection of data is over, an in-built
software extracts meaningful patterns from the gathered data.
This is done using machine learning techniques that process
different modalities speech recognition, natural language
processing, or facial expression detection.
The method devised for the recognition of the facial
expressions is known as the Facial Action Coding System
(FACS). The central concept of the FACS is Action Units
(AU). Action units is basically a list of combination of two
numbers where each set of numbers correspond to the various
human expressions possible This system forms the base of a
complex emotional identification system :
Emotion Action Units
Happiness 6+12
Sadness 1+4+15
Surprise 1+2+5B+26
Fear 1+2+4+5+20+26
Anger 4+5+7+23
9+15+16 where “+” means
Disgust
“and”
By identifying the different facial expressions, we can map
them to their corresponding Action Unit code. Consequently,
they have proposed the following classification of the six
basic emotions, according to their Action Units).
Gestures along with speech and face recognition could be
efficiently used as a means of detecting a particular emotional
state of the user. Depending on the specific action, gestures
could be simple reflexive responses, like lifting our shoulders
when we don’t know the answer to a question, or they could
be complex and meaningful as when communicating with sign
language. Without making use of any object or surrounding
environment, we can wave our hands, clap or point. On the
other hand, when using objects, we can move, touch or handle
those objects. A computer recognizes these, analyzes the input
and responds in a meaningful way.
The advancement in this technology identifies gestures based
on a futuristic 3-D model. Such a model makes use of 3-D
information of key elements of the body parts in order to
obtain several important parameters like palm position or joint
angles.
V. ALGORITHMS
The process of mind reading requires the creation of a reliable
database which is broad enough to fit every need for its
application as well as the selection of an efficient algorithm
classifier which allows for quick and accurate emotion
identification.
Some of the most frequently used algorithm for mind reading
are linear discriminant classifiers (LDC), k-nearest neighbor
(k-NN), artificial neural networks (ANN), Gaussian mixture
model (GMM) amongst many others. It is of utmost
importance that the right classifier is chosen (based on the
type of application) to enhance the overall performance of the
system.
Linear Discriminator classifiers
This is the most widely used classifier. The working of LDC
is described as follows:
If the input feature vector to the classifier is a real vector ,
then the output score is
where a real vector of weights and f is a function that
converts the dot product of the two vectors into the desired
output. The weight vector is learned from a set of labeled
training samples. Often f is a simple function that maps all
values above a certain threshold to the first class and all other
values to the second class. A more complex f gives the
probability that an item belongs to a certain class.
For a two-class classification problem, a linear classifier splits
a high-dimensional input space with a hyperplane: all points
on one side of the hyperplane are classified as "yes" while the
others are classified as "no". A linear classifier is often used in
situations where the speed of classification is an issue such as
face detection software since it is often the fastest classifier.
Support vector machine
Another most commonly used classifier is Support vector
machine (SVM) An SVM is a discriminative classifier
formally defined by a separating hyperplane. The algorithm
outputs an optimal hyperplane by considering the distance of
the 2D points from the hyperplane.
Figure2. Multiple hyperplane
Steps
 • For a linearly separable set of 2D-points which
belong to one of two classes, find a separating
straight line.
• A line is bad if it passes too close to the points
because it will be noise sensitive and it will not
generalize correctly. Therefore, our goal should be to
find the line passing as far as possible from all
points.
• Then, the operation of the SVM algorithm is based
on finding the hyperplane that gives the largest
minimum distance to the training examples.

Figure4. Haemodynamic Response function graph

VII. ENHANCEMENTS
• Imagine being able to tweet what you are thinking
without having to pick up your phone, key in your
message and hit send.
• Also, by just thinking ‘Facebook’ your screen will
show a summary of your friend’s activity onscreen.
• This technology shifts the action from imaginary
Figure3. Optimal hyperplane for the input graph world of science fiction firmly into the scientific
realm.

VI. TECHNOLOGY INVOLVED CONCLUSION

This technology plays an important in Brain-Computer
Interface. Functional Near-Infrared Spectroscopy (fNIR or
fNIRS), is the use of near infrared spectroscopy for the
purpose of neuroimaging. Using fNIR, brain activity is
measured through hemodynamic responses associated with
neuron behavior where in response to performing physical
activities, the body must adjust its blood flow in order to
deliver nutrients such as oxygen and glucose to stressed
tissues that allows them to function. Infra-red light is shone on
the skin by special sensor hardware, and the amount of light
reflected is measured. The amount of reflected and transmitted
light correlates to the BVP as light is absorbed by hemoglobin
which is found richly in the blood stream. By measuring
subject’s Blood Volume Pulse (BVP) and hemoglobin in the
blood by a process called photoplethysmography, a graph is
generated which indicates the blood flow through the
extremities. The peaks of the waves indicate a cardiac cycle
where the heart has pumped blood. If the subject experiences
fear or is startled, their heart usually ‘jumps’ and beats quickly
for some time, causing the amplitude of the cardiac cycle to
increase. This can clearly be seen on a photoplethysmograph
when the distance between the trough and the peak of the
wave decreases. As the subject calms down, and as the body’s
inner core expands, allowing more blood to flow back to the
extremities, the cycle will return to normal.
The human-computer interaction via mind reading computers
is a technology which is still in the beta stage. It promises to
completely change the way we interact with computers by
giving us a totally new and advanced platform for interaction.
The advancements in this technology are moving at a rapid
pace and the day is not far when the terms keyboard and
mouse will sound too obsolete.
REFERENCES
[1] www.wikipedia.org
[2] www.seminaronly.com
[3]The emperor’s new mind by Roger Penrose
[4]http://docs.opencv.org/doc/tutorials/ml/introduction_to_sv
m/introduction_to_svm.html
[5]http://faceinsights.com/downloads/PhysiognomyStudioSetu
p.exe