Curr Psychiatry Rep (2014) 16:523
DOI 10.1007/s11920-014-0523-3
PSYCHIATRY IN THE DIGITAL AGE (JS LUO, SECTION EDITOR)
New Measures of Mental State and Behavior Based on Data
Collected From Sensors, Smartphones, and the Internet
Tasha Glenn & Scott Monteith
Published online: 12 October 2014
# Springer Science+Business Media New York 2014
Abstract With the rapid and ubiquitous acceptance of new
technologies, algorithms will be used to estimate new mea-
sures of mental state and behavior based on digital data. The
algorithms will analyze data collected from sensors in
smartphones and wearable technology, and data collected
from Internet and smartphone usage and activities. In the
future, new medical measures that assist with the screening,
diagnosis, and monitoring of psychiatric disorders will be
available despite unresolved reliability, usability, and privacy
issues. At the same time, similar non-medical commercial
measures of mental state are being developed primarily for
targeted advertising. There are societal and ethical implica-
tions related to the use of these measures of mental state and
behavior for both medical and non-medical purposes.
Keywords Remote monitoring . E-mental health .
Smartphone . Behavioral targeting . Emotion recognition
Introduction
Today, psychiatrists rely on patient history and observation,
objective data such as laboratory tests, and clinical judgment
to diagnose mental illness and assess treatment. There is
renewed emphasis on finding additional objective measures
This article is part of the Topical Collection on Psychiatry in the Digital
Age
T. Glenn
ChronoRecord Association, Inc., Fullerton, CA 92834, USA
S. Monteith (*)
Michigan State University College of Human Medicine Traverse
City Campus, 1400 Medical Campus Drive, Traverse City,
MI 49684, USA
e-mail: monteit2@msu.edu
to assist with these challenges [1, 2]. A biomarker is an
objective measure of a normal biological process, pathogenic
processes, or response to a therapeutic intervention [3, 4].
Categories of potential psychiatric biomarkers include genet-
ic, proteins or other molecules, or neuroimaging findings [1, 5,
6]. With the rapid emergence and acceptance of digital tech-
nologies, alternative measures of mental state and behavior are
being developed for screening, diagnosis, and monitoring. In
contrast to symptoms reported by patients directly to psychi-
atrists, these measures will be based on data collected from
diverse sensors within smartphones, devices, and wearable
technology, as well as data collected from smartphone and
Internet usage and activities. Algorithms will be used to
analyze the collected data to estimate mental health and be-
havior. In the future, the new measures of mental state and
behavior will be available not only from regulated products
with scientifically demonstrated clinical utility, but also from
unregulated applications aimed at consumers and businesses.
Regardless of whether these alternative measures are consid-
ered biomarkers, the new measures are coming and will
impact psychiatry.
The Coming Age of Pervasive Healthcare
Ubiquitous computing and ubiquitous communications are
behind the upcoming measures of mental state and behavior.
In 1991, Mark Weiser of Xerox PARC noted that “the most
profound technologies are those that disappear” and predicted
that computing elements would become so ubiquitous that no
one would notice their presence [7]. We are quickly moving
toward that vision [8] as about 5 % of man-made objects now
contain embedded microprocessors [9]. Cell phones are
viewed as such an essential part of life that many feel uncom-
fortable or inadequate without them [10, 11]. Ubiquitous
communications, or the ability to communicate anytime and
523, Page 2 of 10
anywhere, was enabled by the recent expansion of wireless
communications and ad hoc networking [12]. By 2020, Cisco
Systems estimates that there will be 50 billion devices and
objects connected to the Internet [13] for a world population
estimated to be 7.5 billion [14]. One predicted result of this
extraordinary growth in computing is pervasive healthcare or
the access to healthcare applications for anyone, anytime [12,
15, 16]. A major component of pervasive healthcare will be
remote disease monitoring, including mental state and behav-
ior [15, 16], with the US market for remote disease monitoring
projected to reach $22.5 billion by 2015 [17].
Potential Benefits for Psychiatry
Although the majority of remote disease monitoring efforts
are focused upon somatic illnesses, research into new mea-
sures and monitoring for psychiatry is increasing. New mea-
sures may help to prevent the frequent long time lag between
symptom onset and diagnosis, which is of particular impor-
tance as earlier intervention may lead to a better outcome in
schizophrenia and bipolar disorder [18–20]. There is a need
for unobtrusive, easy-to-use, and inexpensive technologies to
provide objective information about the symptoms that pa-
tients frequently do not report accurately. For example, about
40–70 % of patients do not adhere with treatment regimens for
psychiatric medications [21, 22]. Psychiatric medications con-
taining ingestible sensors will provide data on exactly when
the medications were taken [23•]. There is only moderate
correlation between self-reported and objective measures of
sleep [24], and technologies will provide an understanding of
sleep behaviors and patterns. Some objective measures will
assist with diagnostic challenges due to overlapping symp-
toms and similar clinical presentations of many psychiatric
disorders [25]. Other objective measures will help track the
progression of gradually changing symptoms like cognition,
and technology may help some patients with cognitive im-
pairment to live independently [26, 27].
Basis for New Measures for Psychiatry
Many measures are currently being investigated by medical
researchers for use in psychiatry, including analysis of speech,
facial expression, gesture, posture, movement, eye tracking, as
well as Internet and smartphone behavior and activity. Two
general classes of data are fundamental to these measures of
mental state and behavior: (1) data collected from sensors and
(2) data collected from Internet and smartphone activities.
Algorithms are used to estimate the measures based on the
collected data.
Curr Psychiatry Rep (2014) 16:523
Data Collected From Sensors
The embedded computing element most critical to the expan-
sion of pervasive healthcare computing is sensors. A wide
range of intelligent, lightweight, ultra-low-power, and low-
cost sensors are now available for medical monitoring [28,
29]. These sensors are embedded in smartphones; are wear-
able (embedded in clothing), implantable, and ingestible; and
are placed in the home including in walls and floors [30–32].
A typical smartphone contains multiple sensors such as inter-
nal motion (accelerometer), ambient light, gyroscopic, ges-
ture, magnetometer, temperature and humidity, and barometer.
The communication interfaces commonly found on a
smartphone are Wi-Fi, GPS, near field communications
(NFC), Bluetooth, and infrared (IR) LED. Many additional
sensors are available for physiological measurements [33].
Research into the use of physiologic sensors to measure
emotion is increasing, including skin conductivity, heart rate,
respiration, blood pressure, ECG, EEG, and EMG [34].
Data Collected From Internet and Smartphone Activities
Measures of smartphone activities that are input into medical
algorithms include details on incoming and outgoing call
frequency, duration, and contacts [35]. Measures of Internet
activities include details on search queries, pages visited,
website type, advertising selected, and e-commerce history
[36]. The content of user-created data such as emails, SMS
(text messages), social media, or blogs can be analyzed [37•].
Additionally, metadata (information about information) are
created for transactions from a smartphone or the Internet,
which includes account numbers, login IDs, passwords,
browser types, IP addresses, web pages visited, date, time,
email sender and recipient, cookies, and device fingerprints
[38].
Potential New Measures for Psychiatry
Table 1 provides examples of medical research for psychiatry.
These measures have the potential to provide similar benefits
to psychiatry that are associated with traditional biomarkers in
relation to screening, diagnosis, and monitoring. Many of
these systems are prototypes, and measures of emotion and
behavior discovered in controlled medical or research condi-
tions may not be suitable for natural settings. Future measures
that are able to successfully integrate multiple modalities, such
as speech and facial expression, are expected to be more
precise at identifying emotion and behavior [39].
Additionally, combining the results of multiple measurements
may increase diagnostic accuracy [40].
Curr Psychiatry Rep (2014) 16:523 Page 3 of 10, 523

Table 1 Examples of medical projects in development to measure mental health for psychiatry

Study Year Technology Item measured Aim

Chang [110] 2011 Cell phone or smartphone Speech analysis Mental health monitor of emotion and stress
Dickerson [111] 2011 Multiple sensors and devices Multiple, including speech, activity, Continuous monitoring for depression, PTSD
in home sleep, weight, movement
Roh [112] 2012 Headband system with embedded EEG and heart rate variability Monitor mental stress level
electrodes
Vidal [113] 2012 Wearable eye trackersa Eye tracking and eye movement Mental health monitoring; discriminate among
disorders
Minassian [114] 2010 Wearable vest with integrated Motor activity levels, using Define activity patterns in mania and
sensorsb accelerometers and ECG schizophrenia
Gruenerbl [115] 2014 Smartphone acceleration and Activity and mobility Mood state recognition (mania and depression
GPS sensors in bipolar disorder)
Kane [23•] 2013 Ingestible sensor in tabletsb Medication ingestion Monitor medication adherence in
schizophrenia or bipolar disorder
Kappeler-Setz [116]2013 Wearable sock with sensor Electrodermal activity Monitor patients with bipolar disorder
of skin conductance (sweat secretion)
Jashinsky [117] 2014 Twitter Keywords and phrases in tweets Surveillance for suicide risk
De Choudhury [37•] 2013 Twitter Usage patterns, language Predict onset of depression
Kotikalapudi [36] 2012 Internet usage Traffic volume, type of activities Passive monitoring for depression
and sites
Alvarez-Lozano [35] 2014 Smartphone Usage patterns, app selection Monitor mood in bipolar disorder
McIntyre [118] 2009 Video images Facial activity and expression Identify depression and anxiety
Matic [119] 2012 RFID in clothing, video Dressing ability Monitor cognitive skills
Miskelly [120] 2005 Smartphone with GPSa Location Track wandering in dementia
a
Using commercially available products
b
Using commercially available, FDA-approved products

Establish Clinical Utility Patient Issues

As with any candidate biomarker, all new measures of mental
state and behavior need to be thoroughly evaluated for use in
routine clinical practice. To have clinical utility, the measure
should add new information or clinical value to the informa-
tion that is already available, and should be cost-effective [41].
Depending on the planned role of the measure such as screen-
ing, diagnosis, or monitoring, evaluation may include tests of
reliability (repeatability in the same and different settings),
validity (sensitivity, specificity, and predictive values), dis-
crimination (receiver operator curves), association (relative
risk), and cost-effectiveness analysis [41].
The process to establish clinical utility for biomarkers can
be long and difficult [41, 42]. Initial studies of new measures
often use small, highly selected samples, and results may not
be generalizable to other clinical samples [42]. These small
studies may also overrepresent the prevalence of the disease as
compared with the general population, and results of validity
analyses vary with the prevalence [43]. Confounders that may
alter the results of the measurement need to be identified for
each planned role of the measure [44]. Additionally, new
diagnostic or predictive measures that are correlated with
currently available measures may not provide additional clin-
ical benefit [40, 45].
Patient Technical Skill
Many people with mental illness may not have regular access
to smartphones or the Internet or may lack the technical skills
to use these effectively [46]. Usability of the monitoring
device is critical, and the patient should receive detailed
training. The product should be simple to operate and main-
tain, and designed for the technical skill level of an average
patient who suffers from the targeted mental illness [12]. All
home monitoring systems require some effort and discipline,
and patient errors are a problem in general medical monitoring
systems, such as inadequately recharging smartphone sys-
tems, wearing sensors incorrectly, and turning connections
off [47•, 48, 49].
Patient Interest Level
The successful use of remote monitoring devices is complete-
ly dependent on the cooperation of the patient. While some
patients find that remote monitoring allows them to feel more
control over their illness, others may resist even if they can
easily use the technology. Some patients with chronic illness
feel overwhelmed by their treatment burdens, do not want the
523, Page 4 of 10
added responsibility of self-monitoring [50], or prefer not to
have a constant reminder that they are ill [51]. Some patients
will become annoyed by the effort involved or lose interest
and stop monitoring. Other patients may be unable to cope
with the routine technological issues when experiencing
symptoms [52].
Technology Issues
Sensor Issues
There are reliability problems with remote disease monitoring
related to data quality and transmission delays [53]. The
limiting characteristics of sensors include battery life, memory
capacity, and difficulties with the patient-sensor interface [54,
55]. With wearable sensor systems, there are problems related
to user comfort, sensor location in garments, quality of skin
contact, and power consumption [53, 55]. Many wearable
sensor systems were designed for use by the elderly, and the
preferences of those with mental illnesses need to be consid-
ered. Unlike in a healthcare setting, remote measurements will
not be repeated by a technician if a reading is invalid. In
addition to sensor malfunctions and communication prob-
lems, real-world user activities can interfere with interpreta-
tion of collected sensor data. For example, motion artifacts can
impact the quality of heart rate and respiration measurements
[47•], sweat from exercise can require recalibration of body
worn sensors [31], and ambient noise can impact smartphone
voice analysis [56]. Walls and other obstructions in homes can
interfere with wireless communications.
Data collected from remote monitoring systems can be
obtained at irregular time intervals, often in asynchronous
bursts and in different contexts. The software must be able
to differentiate hardware and software malfunctions from
medical abnormalities. Current algorithms to analyze data
from remote physiological sensors focus on anomaly detec-
tion, prediction, and decision making, especially when used
for continuous monitoring, and many issues remain [57].
Internet and Smartphone Issues
There are also concerns about the algorithms used to estimate
mental state based on Internet and smartphone data and activ-
ities. People vary greatly in their mobile habits, and assess-
ment algorithms based primarily on smartphone usage may
not work well for light smartphone users [58]. Analysis of data
from social media such as Twitter, Facebook, and blogs must
find a few relevant messages among the hundreds of millions
of messages sent daily [59]. The content of these messages
and other social media is noisy unstructured text that varies
tremendously in quality, and automatic text analysis
Curr Psychiatry Rep (2014) 16:523
algorithms are used to separate the high-quality content [60].
Since different methodologies are used to filter, remove noise,
parse and identify emotions, and create user profiles from
social media [37•, 59, 61, 62•], inconsistent results would be
expected among vendors. Additionally, algorithms combine
various additional data elements with the social media to
improve results. Many accounts on social media are fake,
including an estimated 67 million accounts on Facebook
[63] and 20 million accounts on Twitter in 2013 [64].
Finally, not all demographic groups use social media with
the same frequency. In a national US survey in 2013, 67 %
of all adult Internet users accessed any social networking site,
with those aged 18–29 the most likely to do so (83 %) [65].
Context Awareness and Data Interpretation
A critical issue in the interpretation of data for monitoring of
mental health is the user’s context. The implicit situational
information found in human-human interactions is not auto-
matically present in human-computer interactions. This con-
textual awareness can lead to a change in interpretation of
data, such that the relevancy and importance of the informa-
tion depends on the situation, and should be included in all
algorithms [66–68]. As an example, one software framework
to identify context establishes three entities (people, places,
and things) that are described by four categories (identity,
location, status or activity, and time) [68]. Both indoor and
outdoor location sensings are now possible using
smartphones, wristwatches, and sensors [69]. With context-
aware software, the presentation of information, execution of
services, and tagging of data elements will vary with the
context [68]. While context awareness should improve the
quality and clinical usefulness of mental health monitoring,
the required convergence of simultaneous measurements such
as of activity and vital signs will also increase both the volume
of data collected and the processing requirements [66].
Security and Privacy
The security and privacy aspects of systems for monitoring
mental state and behavior are critical for public acceptance
and an area of considerable concern [70, 71]. The medical data
must be accurate and reliable, accessible only by those autho-
rized to do so, and comply with all state and federal privacy
regulations including HIPAA. In addition to the complex
security challenges associated with medical data, there are
many inherent security risks with mobile users, wireless net-
works, shared resources, and shared control of monitoring
systems [47•, 53, 70, 72]. Most breaches of medical data
involve mobile devices [73], and there is a very high risk for
medical data on smartphones or laptops to reach unauthorized
Curr Psychiatry Rep (2014) 16:523
users. In 2013, 3.1 million smartphones were stolen, and only
54 % of smartphone users either set a PIN code to lock the
screen or used encryption [74]. In a 2008 study of 106 US
airports, 12,255 laptops were lost per week, and 55 % of
travelers used no security protections [75]. Monitoring sys-
tems must also be designed and operated to address the threat
of intentional attacks [47•].
Large Influx of New Data
The information flowing from the sensors has the po-
tential to create a massive volume of data [71]. The
scalability of the monitoring systems in terms of both
processing power and volume of data to be stored and
analyzed will become a critical issue as large numbers
of patients start to use these systems [31, 53]. Systems
that function in clinical environments will require an IT
organization to receive and process the patient data and
transfer the results to the psychiatrists.
Measures for Non-medical Commercial Uses
There is widespread non-medical commercial research into
computer recognition of human emotion and behavior. The
future of human-computer interaction envisions flexible
computers that are capable of identifying and tracking
people and then adapting and responding to user moods,
preferences, and intentions [76–79]. Commercial research
to recognize emotion and behavior is focused on develop-
ing subtle, continuous, real-time, and context-specific in-
terpretations of human affective displays and on combin-
ing multiple modalities to improve results [79, 80, 81•,
82]. A range of commercial applications for computer
recognition of human emotion and behavior are
envisioned. These include improving the effectiveness of
targeted advertising [83–85], improving the human-
computer interface for standard devices such as home
appliances [77, 86], developing robotic devices that
can have human-like interactions [79, 87], expanding
the use of computers in tedious but attention-requiring
tasks such as surveillance work [76, 82, 87], recogni-
zing human emotions to improve safety such as when
driving [88], for intelligent tutors in education [89] and
for entertainment.
Table 2 provides examples of non-medical commercial
emotion- and behavior-related research, although much of
this is proprietary. Other technologies being developed
include EmotionML, a proposed standard for an emotion
markup language to allow software to respond to the
detected emotional state of the user from the World
Wide Web consortium (W3C) [90], and the Computer
Page 5 of 10, 523
Expression Recognition Toolkit (CERT) for classifying
emotions from facial expressions [91]. Some products to
analyze human emotion are currently on the marketplace.
Emotient and Affdex based on facial expression analysis,
and Moodies based on voice analysis, are currently being
sold to businesses to analyze customer emotional response
[92–94]. NeuroSky and Emotiv, based on EEG, are cur-
rently being sold to consumers to optimize brain fitness
[95, 96].
Ethical and Societal Issues
Concerns With Medical Measures for Psychiatry
There are many ethical and societal questions associated with
the medical measurements of mental state and behavior.
Psychiatric profiling will identify some individuals as being
at high risk and requiring special intervention. This is espe-
cially troubling when children are involved, as a high-risk
label may shape individual self-perception and societal atti-
tudes about a child [1]. A label of being at high risk for mental
illness could also lead to discrimination by employers,
lenders, insurance companies, and even volunteer organiza-
tions [97].
Patients using remote monitoring should have a clear un-
derstanding of how to obtain help in emergency situations, and
of the standard turnaround time for a response to incoming
messages or data [98]. Some patients with a remote monitor
may incorrectly expect an immediate response to their digital
transmissions whenever they are in trouble [98]. If a patient’s
daily activities are being monitored, the patients should agree to
which specific activities are included [99]. There are additional
ethical considerations regarding the use of surveillance tech-
nologies for individuals with cognitive impairment [100].
Concerns With Measures for Non-medical Commercial Use
There are many areas of concern related to the non-medical
commercial estimates of mental state and behavior. Physicians
are obligated to act in the best interests of their patients, but
commercial application vendors and data brokers are not
involved in patient care and have no such obligation.
The vendors of most applications and devices sold to
consumers do not need to demonstrate efficacy or safety
[101], as these products are not regulated by the FDA
[102]. Many consumers may not be aware that data
from applications not directly accessed by medical pro-
viders are also outside of the scope of HIPAA protec-
tions and may be sold to third parties [103]. Users may
also not understand the sensitivity of some data, such as
from the commercial EEG products, which can be used
as a unique personal identifier [104].
523, Page 6 of 10 Curr Psychiatry Rep (2014) 16:523

Table 2 Examples of non-medical commercial projects in development to measure mood, emotion, and behavior

Study (patent application) Year Technology Item measured Aim

LiKamWa [121] (Microsoft, etc.) 2011 Smartphone SMS (text messaging), calls, email, Measure mood
application usage, web browsing,
location
Chittaranjan [122] (Nokia, etc.) 2011 Smartphone Call logs, SMS, Bluetooth scans, Classify personality traits
application usage
Lu [123•] (Intel, etc.) 2012 Smartphone Stress in voice Classify stress in real-time conversation
Konopnicki [62] (IBM) 2013 Social media Social media data Define user profiles for targeted advertising
Pandey [124] (Yahoo!, 2011 Internet Online queries, browsing, ad clicks, Improve user profiles for behavioral
Google, etc.) history targeting
Aly [125] (Yahoo!) 2012 Internet Internet activities Update profiles daily for behavioral
targeting
Gallager [126] (Raytheon) 2013 Internet Social networking and smartphones Track movement and predict future
behavior
Taigman [127•] (Facebook, etc.) 2014 Internet Facial recognition Human accuracy in facial recognition
Sahami Shirazi [128] 2013 Smartphone Interaction with mobile phone app Track sleep behavior
(Google, etc.)
Samsung [129] 2012 Smartphone Classify social network user Control interactions between users
emotional states related to emotional states
Apple [130] 2014 Hardware/software Voice, facial expression, physiologic, Infer mood and deliver mood-based content
devices Internet activity, compared to a user
baseline profile
Microsoft [131] 2014 Microcontroller Biometric data on emotional state such Mood-actuated device reacts to user mood
as heart rate, skin conductance
Microsoft [85] 2012 Xbox Scan email, messages, Kinect movement Target ads based on emotional state
sensors, facial expressions
Yahoo [132] 2013 Internet Generates user profile with mood gradient, Target items (song playlist, movies) to mood
friends preferences
Yahoo [133] 2014 Mobile devices, Voice analysis, speech, tone Stream content using voice-based mood
computers analysis
Dell [134] 2014 Computer with EEG, heart rate, other physiologic Determine mood and emotion for use in
sensors sensors education, gaming

Commercial organizations that monitor people’s everyday
health-related habits, along with their daily activities, can
combine this with other data obtained from data brokers to
estimate mental state and behavior. Initially, the most frequent
use of the estimates of emotional state and behaviors may be
to profile individuals for sales purposes [105]. However, there
are many possible implications if commercial organizations
define and detect profiles associated with mental illnesses.
Without their knowledge or active participation, people
could be branded and blacklisted by an algorithm.
Targeted advertising for health-related products may in-
fluence medical choices [106] and contain incorrect
content that may mislead, offer false hope of cure
[107], or result in delays in seeking established treat-
ment. Furthermore, the algorithms used by the commer-
cial companies for health analyses are not published,
may be incorrect, and cannot be duplicated [108]. If
these algorithms are considered trade secrets, legal pro-
tections will suppress this information from the public,
even though building on the published findings of
others is fundamental to scientific inquiry [109]. Public
discussions of commercial use of mood, emotion, and
behavioral data are required.
Conclusions
New measures of mental state and behavior that are generated
using technology and analytics are coming to psychiatry. In
the future, clinically useful measures will help with the screen-
ing, diagnosis, and treatment of psychiatric disorders, and
there will be continued advances in this emerging field.
There are many technical issues to resolve, primarily relating
to reliability, usability, privacy, and clinical utility of the new
measures. At the same time, the use of non-medical commer-
cial products based on similar measures will become perva-
sive. Society must address the ethical issues associated with
measures of mental state and behavior for medical and non-
medical use.
Curr Psychiatry Rep (2014) 16:523
Compliance With Ethics Guidelines
Conflict of Interest Scott Monteith declares no conflict of interest.
Tasha Glenn shares a patent for ChronoRecord software but does not
receive any financial compensation from the ChronoRecord Association,
a 501(c)(3) nonprofit organization.
Human and Animal Rights and Informed Consent This article does
not contain any studies with human or animal subjects performed by any
authors.
References
Papers of particular interest, published recently, have been
highlighted as:
• Of importance
1. Singh I, Rose N. Biomarkers in psychiatry. Nature. 2009;460:
202–7.
2. Kapur S, Phillips AG, Insel TR. Why has it taken so long for
biological psychiatry to develop clinical tests and what to do about
it? Mol Psychiatry. 2012;17:1174–9.
3. Biomarkers Definitions Working Group. Biomarkers and surro-
gate endpoints: preferred definitions and conceptual framework.
Clin Pharmacol Ther. 2001;69:89–95.
4. Lesko LJ, Atkinson Jr AJ. Use of biomarkers and surrogate end-
points in drug development and regulatory decision making: criteria,
validation, strategies. Annu Rev Pharmacol Toxicol. 2001;41:347–
66.
5. Lakhan SE, Kramer A. Schizophrenia genomics and proteomics:
are we any closer to biomarker discovery? Behav Brain Funct.
2009;5:2.
6. McIntyre RS, Cha DS, Jerrell JM, et al. Advancing biomarker re-
search: utilizing ‘Big Data’ approaches for the characterization and
prevention of bipolar disorder. Bipolar Disord. 2014;16(5):531–47.
7. Weiser M. The computer for the 21st century. Sci Am. 1991;265:
94–104.
8. Gershenfeld N, Krikorian R, Cohen D. The Internet of things. Sci
Am. 2004;291:76.
9. Vinge V. Who’s afraid of first movers? Singularity Summit 2012.
https://docs.google.com/document/d/1W7Vs8e3MNwjU_
1ZZZqmq0O-dFKCwD4VRyww-qOvDdRU/pub. Accessed 9
Jul 2014.
10. Venta L, Isomursu M, Ahtinen A et al. “My phone is a part of my
soul”—how people bond with their mobile phones. In Mobile
Ubiquitous Computing, Systems, Services and Technologies,
2008. UBICOMM ’08. The second international conference on
IEEE, 2008; 311–17
11. Srivastava L. Mobile phones and the evolution of social behaviour.
Behav Inf Technol. 2005;24:111–29.
12. Korhonen I, Bardram JE. Guest editorial: introduction to the
special section on pervasive healthcare. IEEE Trans Inf Technol
Biomed. 2004;8:229–34.
13. Cisco Systems. Internet of things. http://www.cisco.com/web/
solutions/trends/iot/indepth.html. Accessed 9 Jul 2014.
14. US Census. International data base. World population: 1950–
2050. 2011 http://www.census.gov/population/international/data/
idb/worldpopgraph.php. Accessed 9 Jul 2014.
15. Varshney U. Pervasive healthcare and wireless health monitoring.
Mob Netw Appl. 2007;12:113–27.
Page 7 of 10, 523
16. Orwat C, Graefe A, Faulwasser T. Towards pervasive computing
in health care—a literature review. BMC Med Inform Decis Mak.
2008;8:26.
17. Kalorama Information. Remote and wireless patient monitoring
markets. 2011. http://www.kaloramainformation.com/Remote-
Wireless-Patient-6487095/. Accessed 9 Jul 2014.
18. Marshall M, Rathbone J. Early intervention for psychosis.
Schizophr Bull. 2011;37:1111–4.
19. Berk M, Brnabic A, Dodd S, et al. Does stage of illness impact
treatment response in bipolar disorder? Empirical treatment data
and their implication for the staging model and early intervention.
Bipolar Disord. 2011;13:87–98.
20. Killackey E, Yung AR. Effectiveness of early intervention in
psychosis. Curr Opin Psychiatry. 2007;20:121–5.
21. Bulloch AG, Patten SB. Non-adherence with psychotropic medi-
cations in the general population. Soc Psychiatry Psychiatr
Epidemiol. 2010;45:47–56.
22. Cramer JA, Rosenheck R. Compliance with medication regimens for
mental and physical disorders. Psychiatr Serv. 1998;49:196–201.
23.• Kane JM, Perlis RH, DiCarlo LA, et al. First experience with a
wireless system incorporating physiologic assessments and direct
confirmation of digital tablet ingestions in ambulatory patients
with schizophrenia or bipolar disorder. J Clin Psychiatry.
2013;74:e533–40. Pilot study using sensors embedded in placebo
tablets.
24. Lauderdale DS, Knutson KL, Yan LL, et al. Self-reported and
measured sleep duration: how similar are they? Epidemiology.
2008;19:838–45.
25. Cook IA. Biomarkers in psychiatry: potentials, pitfalls, and prag-
matics. Prim Psychiatry. 2008;15:54.
26. Wood A, Stankovic JA, Virone G, et al. Context-aware wireless
sensor networks for assisted living and residential monitoring.
IEEE Netw. 2008;22:26–33.
27. Braddock D, Rizzolo MC, Thompson M, et al. Emerging technolo-
gies and cognitive disability. J Spec Educ Technol. 2004;19:49–56.
28. Milenković A, Otto C, Jovanov E. Wireless sensor networks for
personal health monitoring: issues and an implementation.
Comput Commun. 2006;29:2521–33.
29. Darwish A, Hassanien AE. Wearable and implantable wireless
sensor network solutions for healthcare monitoring. Sensors.
2011;11:5561–95.
30. Pentland A. Healthwear: medical technology becomes wearable.
Computer. 2004;37:42–9.
31. Alemdar H, Ersoy C. Wireless sensor networks for healthcare: a
survey. Comput Netw. 2010;54:2688–710.
32. Yoo H-J. Your heart on your sleeve: advances in textile-based
electronics are weaving computers right into the clothes we wear.
IEEE Solid-State Circ Mag. 2013;5:59–70.
33. Lowe SA, Ólaighin G. Monitoring human health behaviour in
one’s living environment: a technological review. Med Eng Phys.
2014;36:147–68.
34. Jerritta S, Murugappan M, Nagarajan R, et al. Physiological
signals based human emotion recognition: a review. In Signal
Processing and its Applications (CSPA), 2011 I.E. 7th
International Colloquium on. IEEE. 2011:410–15
35. Alvarez-Lozano J, Frost M, Osmani V, et al. Tell me your apps and
I will tell you your mood: correlation of apps usage with bipolar
disorder state. ACM proceedings of 7th international conference
on pervasive technologies related to assistive environments. 2014.
36. Kotikalapudi R, Chellappan S, Montgomery F, et al. Associating
Internet usage with depressive behavior among college students.
IEEE Technol Soc Mag. 2012;31:73–80.
37.• De Choudhury M, Counts S, Horvitz E, et al. Predicting depres-
sion via social media. 2013. Microsoft Research. http://research.
microsoft.com/apps/pubs/default.aspx?id=192721. Accessed 9 Jul
2014. Example of analyzing social media to predict mood.
523, Page 8 of 10
38. Guardian. A guardian guide to your metadata. 2013 http://www.
theguardian.com/technology/interactive/2013/jun/12/what-is-
metadata-nsa-surveillance%23meta=0000000. Accessed 9 Jul
2014.
39. Picard RW. Affective computing. Cambridge: MIT; 1997.
40. Pepe MS, Thompson ML. Combining diagnostic test results to
increase accuracy. Biostatistics. 2000;1:123–40.
41. Pletcher MJ, Pignone M. Evaluating the clinical utility of a bio-
marker: a review of methods for estimating health impact.
Circulation. 2011;123:1116–24.
42. Perlis RH. Translating biomarkers to clinical practice. Mol
Psychiatry. 2011;16:1076–87.
43. Brenner H, Gefeller O. Variation of sensitivity, specificity, likeli-
hood ratios and predictive values with disease prevalence. Stat
Med. 1997;16:981–91.
44. Mayeux R. Biomarkers: potential uses and limitations. NeuroRx.
2004;1:182–8.
45. Gerszten RE, Wang TJ. The search for new cardiovascular bio-
markers. Nature. 2008;451:949–52.
46. Monteith S, Glenn T, Bauer M. Searching the internet for health
information about bipolar disorder: some cautionary issues. Int J
Bipolar Disord. 2013;1:22.
47.• Ko JG, Lu C, Srivastava MB, et al. Wireless sensor networks for
healthcare. Proc IEEE. 2010;98:1947–60. Clear overview article.
48. Chan M, Estève D, Fourniols JY, et al. Smart wearable systems:
current status and future challenges. Artif Intell Med. 2012;56:
137–56.
49. Healey J, Nachman L, Subramanian S, et al. Out of the lab and
into the fray: towards modeling emotion in everyday life. In
Pervasive computing. Springer, Berlin, 2010:156–73.
50. May C, Montori VM, Mair FS. We need minimally disruptive
medicine. BMJ. 2009;339:b2803.
51. Mol A. Living with diabetes: care beyond choice and control.
Lancet. 2009;373:1756–7.
52. Luxton DD, McCann RA, Bush NE, et al. mHealth for mental
health: integrating smartphone technology in behavioral
healthcare. Prof Psychol Res Pr. 2011;42:505.
53. Baig MM, Gholamhosseini H. Smart health monitoring systems:
an overview of design and modeling. J Med Syst. 2013;37:9898.
54. Jurik AD, Weaver AC. Remote medical monitoring. Computer.
2008;41:96–9.
55. McAdams E, Krupaviciute A, Gehin C, Grenier E, et al. Wearable
sensor systems: the challenges. In Engineering in Medicine and
Biology Society, EMBC, 2011 Annual international conference of
the IEEE. IEEE, 2011:3648–51
56. Lane ND, Mohammod M, Lin M, et al. Bewell: a smartphone
application to monitor, model and promote wellbeing. In 5th
international ICST conference on pervasive computing technolo-
gies for healthcare. 2011: 23–6.
57. Banaee H, Ahmed MU, Loutfi A. Data mining for wearable
sensors in health monitoring systems: a review of recent trends
and challenges. Sensors (Basel). 2013;13:17472–500.
58. Ma Y, Xu B, Bai Y, et al. Daily mood assessment based on mobile
phone sensing. In Wearable and Implantable Body Sensor
Networks (BSN), 2012 Ninth international conference on. IEEE.
2012: 142–7
59. Alexe B, Hernandez MA, Hildrum KW, et al. Surfacing time-
critical insights from social media. In Proceedings of the 2012
ACM SIGMOD international conference on management of data.
ACM, 2012: 657–60.
60. Agichtein E, Castillo C, Donato D, et al. Finding high-quality
content in social media. In Proceedings of the 2008 International
conference on web search and data mining. ACM, 2008:183–194.
61. Wang W, Chen L, Thirunarayan K, et al. Harnessing Twitter ‘big
data’ for automatic emotion identification. In Privacy, Security,
Risk and Trust (PASSAT), 2012 International conference on and
Curr Psychiatry Rep (2014) 16:523
2012 International conference on social computing (SocialCom).
IEEE. 2012:587–592
62.• Konopnicki D, Shmueli-Scheuer M, Cohen D, et al. A statistical
approach to mining customers’ conversational data from social
media. IBM J Res Dev. 2013;57:14–1. Example of creating user
profiles from social media.
63. Munson L. Facebook: At least 67 million accounts are fake. 2014
http://nakedsecurity.sophos.com/2014/02/10/facebook-at-least-
67-million-accounts-are-fake/. Accessed 9 Jul 2014.
64. Perlroth N. Fake Twitter followers become multimillion-dollar
business. New York Times. 2013 http://bits.blogs.nytimes.com/
2013/04/05/fake-twitter-followers-becomes-multimillion-dollar-
business/?_php=true&_type=blogs&ref=technology&_r=0.
Accessed 9 Jul 2014.
65. Duggan M. The demographics of social media users. 2013. http://
www.pewinternet.org/files/old-media/Files/Reports/2013/PIP_
SocialMediaUsers.pdf. Accessed 9 Jul 2014.
66. Viswanathan H, Chen B, Pompili D. Research challenges in
computation, communication, and context awareness for ubiqui-
tous healthcare. IEEE Commun Mag. 2012;50:92–9.
67. Baldauf M, Dustdar S, Rosenberg F. A survey on context-aware
systems. Int J Ad Hoc Ubiquit Comput. 2007;2:263–77.
68. Dey AK, Abowd GD, Salber D. A conceptual framework
and a toolkit for supporting the rapid prototyping of
context-aware applications. Human-computer interaction.
2001;16:97–166.
69. Welbourne E, Lester J, LaMarca A et al. Mobile context inference
using low-cost sensors. In Location-and Context-Awareness.
Springer, Berlin, 2005:254–63.
70. Li M, Lou W, Ren K. Data security and privacy in wireless body
area networks. Wireless Communications, IEEE. 2010;17:51–8.
71. Sarasohn-Kahn J. Making sense of sensors: how new technologies
can change patient care. California Healthcare Foundation. 2013.
http://www.chcf.org/publications/2013/02/making-sense-sensors.
Accessed 9 Jul 2014
72. Halperin D, Kohno T, Heydt-Benjamin TS, et al. Security and
privacy for implantable medical devices. IEEE Pervasive Comput.
2008;7:30–9.
73. Redspin 2014. Breach report 2013: protected health information
(PHI) http://www.redspin.com/resources/whitepapers-datasheets/
Request-2013-Breach-Report-Protected-Health-Information-PHI-
Redspin.php. Accessed 9 Jul 2014.
74. Consumer Reports. Smart phone thefts rose to 3.1 million last
y e a r, C o n s u m e r R e p o r t s f i n d s . 2 0 1 4 . h t t p : / / w w w.
consumerreports.org/cro/news/2014/04/smart-phone-thefts-rose-
to-3-1-million-last-year/index.htm#. Accessed 9 Jul 2014.
75. Ponemon Institute. Airport insecurity: the case of missing & lost
laptops. 2008. http://www.dell.com/downloads/global/services/
dell_lost_laptop_study.pdf. Accessed 9 Jul 2014.
76. Pantic M, Sebe N, Cohn JF et al. Affective multimodal
human-computer interaction. In Proceedings of the 13th
annual ACM international conference on multimedia.
ACM, 2005; 669–76.
77. Schmidt A, Pfleging B, Alt F, et al. Interacting with 21st-century
computers. IEEE Pervasive Comput. 2012;11:22–31.
78. Pentland A. Looking at people: sensing for ubiquitous and wear-
able computing. IEEE Trans Pattern Anal Mach Intell. 2000;22:
107–19.
79. Sebe N, Cohen I, Gevers T, et al. Multimodal approaches for
emotion recognition: a survey. In Electronic Imaging 2005,
International Society for Optics and Photonics, 2005: 56–67
80. Gunes H, Pantic M. Automatic, dimensional and continuous emo-
tion recognition. Int J Synth Emotions (IJSE). 2010;1:68–99.
81.• Zeng Z, Pantic M, Roisman GI, et al. A survey of affect recogni-
tion methods: audio, visual, and spontaneous expressions. A sur-
vey of affect recognition methods: audio, visual, and spontaneous
Curr Psychiatry Rep (2014) 16:523
expressions. IEEE Trans Pattern Anal Mach Intell. 2009;31:39–
58. Clear survey article.
82. Aggarwal JK, Ryoo MS. Human activity analysis: a review. ACM
Comput Surv. 2011;43:16.
83. Network Advertising Initiative. Study finds behaviorally-targeted
ads more than twice as valuable, twice as effective as non-targeted
online ads. 2010 http://www.networkadvertising.org/pdfs/NAI_
Beales_Release.pdf. Accessed 9 Jul 2014
84. Yan J, Liu N, Wang G, et al. How much can behavioral targeting
help online advertising? In: Proceedings of the 18th international
conference on world wide web. ACM. 2009:261–270.
85. Microsoft, owner. Chung WJ, Patwa P, Markov MM, inventors.
Targeting advertisements based on emotion. United States Patent
Application 20120143693 A1. 6/7/12
86. Meyer S, Rakotonirainy A. A survey of research on context-aware
homes. In Proceedings of the Australasian information security
workshop conference on ACSW frontiers. Australian Computer
Society, Inc., 2003; 21:159–68.
87. Chui M, Löffler M, Roberts R. The internet of things. McKinsey
Quarterly 2. 2010: 1–9.
88. Lisetti CL, Nasoz F. Using noninvasive wearable computers to
recognize human emotions from physiological signals. EURASIP
J Appl Signal Processing. 2004: 1672–87.
89. D’Mello S, Picard R, Graesser A. Towards an affect-sensitive
autotutor. IEEE Intell Syst. 2007;22:53–61.
90. Schröder M, Baggia P, Burkhardt F, et al. EmotionML—an up-
coming standard for representing emotions and related states. In
Affective Computing and Intelligent Interaction. Springer, Berlin.
2011:316–25.
91. Littlewort G, Whitehill J, Wu T et al. The computer expression
recognition toolbox (CERT). In Automatic Face & Gesture
Recognition and Workshops (FG 2011). IEEE International con-
ference on. IEEE. 2011:298–305
92. Emotient. www.emotient.com. Accessed 9 Jul 2014.
93. Affdex. http://www.affdex.com/. Accessed 9 Jul 2014.
94. Moodies. http://www.beyondverbal.com/emotions-analytics/why-
is-it-so-special/. Accessed 9 Jul 2014.
95. NeuroSky. http://neurosky.com/. Accessed 9 Jul 2014.
96. Emotiv Insight. https://www.kickstarter.com/projects/tanttle/
emotiv-insight-optimize-your-brain-fitness-and-per. Accessed 9
Jul 2014.
97. Overton SL, Medina SL. The stigma of mental illness. J Couns
Dev. 2008;86:143–51.
98. Luxton DD, June JD, Kinn JT. Technology-based suicide preven-
tion: current applications and future directions. Telemed J E
Health. 2011;17:50–4.
99. Glascock AP, Kutzik DM. Behavioral telemedicine: a new ap-
proach to the continuous nonintrusive monitoring of activities of
daily living. Telemed J. 2000;6:33–44.
100. Niemeijer AR, Frederiks BJ, Riphagen II, et al. Ethical and prac-
tical concerns of surveillance technologies in residential care for
people with dementia or intellectual disabilities: an overview of
the literature. Int Psychogeriatr. 2010;22:1129–42.
101. Sharpe R. Many health apps are based on flimsy science at best,
and they often do not work. The Washington Post. 2012. http://
www.washingtonpost.com/national/health-science/many-health-
apps-are-based-on-flimsy-science-at-best-and-they-often-do-not-
work/2012/11/12/11f2eb1e-0e37-11e2-bd1a-b868e65d57eb_
story.html. Accessed 9 Jul 2014.
102. FDA. Mobile medical applications. 2014. http://www.fda.gov/
MedicalDevices/ProductsandMedicalProcedures/
ConnectedHealth/MobileMedicalApplications/default.htm?utm_
source=twitterfeed&utm_medium=twitter. Accessed 9 Jul 2014.
103. Glenn T, Monteith S. Privacy in the digital world: medical and
health data outside of HIPAA protections. Curr Psychiatry Rep.
2014;16(11):494.
Page 9 of 10, 523
104. Stopczynski A, Greenwood D, Hansen LK. et al. Privacy for
personal neuroinformatics. 2014. http://arxiv.org/abs/1403.2745.
Accessed 9 Jul 2014.
105. Labrecque LI, Markos E, Milne GR. Online personal branding:
processes, challenges, and implications. J Interact Mark. 2011;25:
37–50.
106. Stark M, Fins JJ. Engineering medical decisions: computer algo-
rithms and the manipulation of choice. Camb Q Healthc Ethics.
2013;22:373–81.
107. Petersen A, Seear K. Technologies of hope: techniques of the
online advertising of stem cell treatments. New Genet Soc.
2011;30:329–46.
108. Lazer D, Kennedy R, King G, et al. Big data. The parable of
Google Flu: traps in big data analysis. Science. 2014;343:1203–5.
109. Simpson MP. The future of innovation-trade secrets, property
rights, and protectionism-an age-old tale. Brook L Rev. 2005;70:
1121.
110. Chang KH, Fisher D, Canny J, et al. How’s my mood and stress?: an
efficient speech analysis library for unobtrusive monitoring on mo-
bile phones. In Proceedings of the 6th international conference on
body area networks. ICST (Institute for Computer Sciences, Social-
Informatics and Telecommunications Engineering), 2011; 71–7.
111. Dickerson RF, Gorlin EI, Stankovic JA. Empath: a continuous
remote emotional health monitoring system for depressive illness.
In Proceedings of the 2nd conference on wireless health. ACM.
2011: 5.
112. Roh T, Bong K, Hong S, et al. Wearable mental-health monitoring
platform with independent component analysis and nonlinear
chaotic analysis. Conf Proc IEEE Eng Med Biol Soc.
2012;2012:4541–4.
113. Vidal M, Turner J, Bulling A, et al. Wearable eye tracking for
mental health monitoring. Comput Commun. 2012;35:1306–11.
114. Minassian A, Henry BL, Geyer MA, et al. The quantitative as-
sessment of motor activity in mania and schizophrenia. J Affect
Disord. 2010;120:200–6.
115. Gruenerbl A, Osmani V, Bahle G, et al. Using smart phone
mobility traces for the diagnosis of depressive and manic episodes
in bipolar patients. In Proceedings of the 5th augmented human
international conference. ACM, 2014:38.
116. Kappeler-Setz C, Gravenhorst F, Schumm J, et al. Towards long
term monitoring of electrodermal activity in daily life. Pers
Ubiquitous Comput. 2013;17:261–71.
117. Jashinsky J, Burton SH, Hanson CL, et al. Tracking suicide risk
factors through Twitter in the US. Crisis. 2014;35:51–9.
118. McIntyre G, Gocke R, Hyett M, et al. An approach for automat-
ically measuring facial activity in depressed subjects. In Affective
computing and intelligent interaction and workshops, 2009. ACII
2009. 3rd International conference on, IEEE, 2009:1–8
119. Matic A, Mehta P, Rehg JM, et al. Monitoring dressing activity
failures through RFID and video. Methods Inf Med. 2012;51:45–
54.
120. Miskelly F. Electronic tracking of patients with dementia and
wandering using mobile phone technology. Age Ageing.
2005;34:497–9.
121. LiKamWa R, Liu Y, Lane ND, et al. Can your smartphone infer
your mood?. In Proc. PhoneSense Workshop. 2011. Microsoft
Research. http://research.microsoft.com/en-us/projects/
moodsense/. Accessed 9 Jul 2014.
122. Chittaranjan G, Blom J, Gatica-Perez D. Who’s who with big-five:
analyzing and classifying personality traits with smartphones. In
Wearable Computers (ISWC), 2011 15th annual international sympo-
sium on, IEEE. Nokia Research Center. https://research.nokia.com/
publications?search=Gokul+Chittaranjan. Accessed 9 Jul 2014
123.• Lu H, Frauendorfer D, Rabbi M, et al. StressSense: detecting stress
in unconstrained acoustic environments using smartphones. In
Proceedings of the 2012 ACM conference on ubiquitous
523, Page 10 of 10
computing. ACM. 2012:351–360. Example of voice analysis
using smartphones.
124. Pandey S, Aly M, Bagherjeiran A, et al. Learning to target: what
works for behavioral targeting. In Proceedings of the 20th ACM
international conference on Information and knowledge manage-
ment. ACM. 2011:1805–14.
125. Aly M, Hatch A, Josifovski V, et al. Web-scale user modeling for
targeting. In Proceedings of the 21st international conference
companion on World Wide Web. ACM. 2012:3–12.
126. Gallagher R. Software that tracks people on social media created
by defence firm. The Guardian. 2013. http://www.theguardian.
com/world/2013/feb/10/software-tracks-social-media-defence?
intcmp=239. Accessed 9 Jul 2014
127.• Taigman Y, Yang M, Ranzato M, et al. DeepFace: closing the gap
to human-level performance in face verification. 2014. https://
www.facebook.com/publications/546316888800776/. Accessed
9 Jul 2014. Advances in facial recognition.
128. Sahami Shirazi A, Clawson J, Hassanpour Y, et al. Already up?
using mobile phones to track & share sleep behavior. Int J Hum
Comput Stud. 2013;71:878–88.
Curr Psychiatry Rep (2014) 16:523
129. Samsung, owner. Lee H, inventor. Apparatus and method for
sharing user’s emotion. United States Patent Application
20130144937 A1. 7 Aug 2012.
130. Apple, owner. Greenzeiger MF, Phulari R, Sanghavi M,
inventors. Inferring user mood based on user and group
characteristic data. United States Patent Application
20140025620 A1. 23 Jan 2014.
131. Microsoft, owner. Roseway A, Davis FA, Carroll EA, et al. inven-
tors. Mood-actuated device. United States Patent Application
20140085181 A1. 27 Mar 2014
132. Yahoo, owner. White W, Slaney M, inventors. System and method
for generating a playlist from a mood gradient. United States
Patent Application US8583615 B2. 12 Nov 2013
133. Yahoo, owner. Kamdar G, inventor. Method and system for voice
based mood analysis. United States Patent Application
20140046660 A1. 13 Feb 2014
134. Hamblem M. Dell’s new research division wants computers to detect
your mood. Computerworld. 2014. http://www.computerworld.com/
s/article/9247285/Dell_s_new_research_division_wants_computers_
to_detect_your_mood. Accessed 9 Jul 2014