2017 Tenth International Conference on Mobile Computing and Ubiquitous Network (ICMU)
Considering Interpersonal Differences in Validating
Wearable Sleep-Tracking Technologies
Zilu Liang
Dept. of Engineering, The University of Tokyo
Hongo 7-3-1, Tokyo, Japan 113-0022
z-liang@t-adm.t.u-tokyo.ac.jp
Abstract—This study investigated an important yet usually
neglected question in validating consumer sleep tracking devices:
does interpersonal difference play a role in the validity of
consumer sleep trackers? Most existing validation studies assume
that the accuracy of these devices is consistent cross individuals,
ignoring the fact that human sleep demonstrates significant
interpersonal differences. This study aimed to test this
assumption through validating two newest gadgets of home sleep
tracking technologies, i.e. Fitbit Charge 2 (wearable wristband)
and Neuroon (wearable EEG eye mask), in comparison to a
clinical device. Two participants were recruited to track their
sleep using a Fitbit, a Neuroon, and a clinical device SLEEP
SCOPE simultaneously for approximately 40 nights. Data
analysis was conducted for each participant separately. The
results showed that the Fitbit measurements completely deviated
from the clinical measurements for one participant, but agreed
very well for another participant, especially in the dimension of
total sleep time. Neuroon measurements agreed well but were
uncorrelated to clinical measurements for one participant, and
were correlated significantly but agreed poorly for another
participant. This study suggested that interpersonal differences
should be considered in future validation studies and in designing
better consumer sleep-tracking technologies.
Keywords—Sleep; wearables; EEG; Fitbit; validation; n-of-1;
personal informatics.
I. INTRODUCTION
As sensing and wearable technologies advance, many home
sleep tracking technologies emerge in consumer market. These
technologies range from free mobile applications (e.g. Sleep As
Android, Sleep Bot) to affordable wearable devices (e.g. Fitbit,
Jawbone) to portable EEG (e.g. Neuroon and Sleep Shepherd).
Since human sleep demonstrates significant interpersonal
differences [1-2], it is an interesting question as to whether
consumer sleep trackers are equivalently accurate for all
individuals. Most of the work in literature assumes that the
accuracy of these devices is consistent across individuals [3].
The aim of this study is to test this assumption through
validating two newest gadgets of home sleep tracking
technologies in comparison to a clinical device. To investigate
interpersonal differences, we adopted the n-of-1 approach (i.e.
personalized approach) [4] in place of the 1-of-n approach (i.e.
large-sample approach) [5] that has been used in most of the
existing studies. We recruited two participants who wore a
Fitbit Charge 2 wearable wristband, a Neuroon wearable EEG
eye mask, and a clinical device named SLEEP SCOPE (Sleep
This study was sponsored by JSPS KAKENHI Grant-in-Aid for Research
Activity Start-up (Grant Number 16H07469). The opinions expressed in
this paper do not represent the views of the second author’s company.
978-4-907626-31-0/17/$31.00 ©2017
Authorized licensed use limited to: MapuaIEEE
University. Downloaded on May 01,2021 at 12:32:40 UTC from IEEE Xplore. Restrictions apply.
Mario Alberto Chapa Martell
CAC Corporation
Hakozaki-cho 24-1 , Tokyo, Japan
Mchapam0300@gmail.com
Well Co., Osaka, Japan) for approximately 40 nights. The
measurements of consumer devices were validated against
those by SLEEP SCOPE and such analysis were done for each
participant separately. We then compared the results obtained
from each of the participants. The measurements by SLEEP
SCOPE were used as the ground truth to compare with.
We used several statistical techniques to analyze the
validity of consumer sleep trackers. The results revealed
significant inter-personal differences. First, the Fitbit
measurements completely deviated from the clinical
measurements for one participant, but agreed very well for
another participant especially in the dimension of total sleep
time (TST). Second, the Neuroon measurements agreed well
but were uncorrelated to the clinical measurements for one
participant (on SOL), and were correlated significantly but
agreed poorly for another participant (on WASO and SE). In
addition, we found that good agreement between Neuroon and
SLEEP SCOPE generally occurred when the user had a good
night of sleep. The main contributions of this study are two-
fold: (1) the results revealed that the validity of consumer
wearable devices varies from user to user; (2) we identified
conditions for good validity of consumer wearable devices to
inform the future design of accurate consumer sleep trackers.
The outcome of this study brought forward an important
question that we need to consider in future validation studies
and in designing better consumer sleep tracking technologies.
II. RELATED WORK
In clinical studies, polysomnography (PSG) and actigraphy
are the most widely used devices for objectively measuring
sleep. It has been most useful for the diagnosis and treatment of
sleep diseases such as obstructive sleep apnea (OSA),
narcolepsy, rapid-eye-movement sleep behavior disorder [6].
Actigraphy, on the other hand, is mostly used for the diagnosis
of circadian rhythm disorders [7].
In addition to the clinical sleep monitoring devices, many
consumer sleep tracking devices has emerged over recent years.
These technologies can be generally divided into two
categories, i.e. sleep tracking devices based on movement (e.g.
Fitbit, Jawbone, mobile applications) and sleep tracking
devices based on brain activity signals (e.g. Zeo headband,
Neuroon eye mask, Sleep Shepherd headband) [14-15]. This
field is expanding rapidly and new sleep tracking devices are
introduced to the consumer market every year. Validation
studies on previous versions of Fitbit and Jawbone found that
 2017 Tenth International Conference on Mobile Computing and Ubiquitous Network (ICMU)

Fitbit devices overestimated TST and SE and underestimated participants are shown in Table II (according to the
WASO compared to PSG and clinical actigraphy in “normal measurements of SLEEP SCOPE).
mode”. While in “sensitive mode”, Fitbit devices substantially TABLE II. BASIC STATISTICS OF SLEEP QUALITY
underestimated TST and SE and overestimated WASO [8-9].
In this study, we aim to investigate the impact of interpersonal TST WASO SOL
NAWK SE (%)
(min) (min) (min)
differences on the accuracy of recently-released sleep trackers Participant
in the consumer market, i.e. Fitbit Charge 2 wristband and 1
408േ30 34േ11 21േ14 34േ8 88േ4
Neuroon wearable EEG eye mask, with the purpose of Participant
412േ40 34േ33 8േ7 27േ8 90േ7
establishing a baseline for the state-of-the-art in this field. 2
TABLE I. DEFINITION OF MEASURED SLEEP METRICS TABLE III. Z-VALUES OF WSR TEST ON SLEEP METRICS

Metrics Definition TST WASO SOL NAWK SE

Time in minutes from sleep onset to sleep
Total Sleep Time (TST)
offset.
Wake after Sleep Onset Periods of wakefulness occurring after
(WASO) defined sleep onset.
Sleep Onset Latency Time in minutes from “light out” to the
(SOL) first epoch scored as sleep.
The number of awakenings occurring after
Awake Count
defined sleep onset.
Percentage of total time in bed actually
Sleep Efficiency (SE)
spent in sleep. TST/(TST+WASO)
III. METHOD
Two participants without sleep problems were recruited for
collecting sleep data (Participant 1: sex=F, age=28; participant
2: sex=M, age=32). Each participant wore a Fitbit Charge 2 on
the non-dominant wrist and a Neuroon wearable EEG eye
mask on the lower forehead. The two electrodes of the clinical
device SLEEP SCOPE were attached to the upper forehead and
behind ear respectively. These devices were used
simultaneously to measure the sleep metrics summarized in
Table I in which the definition of the metrics are also presented
[10]. We used Bland-Altman plot [11], which compares the
measurements’ differences to the measurements’ mean, to
evaluate the agreement between the consumer sleep trackers
(i.e., Fitbit and Neuroon) and the clinical sleep monitor
(SLEEP SCOPE). Wilcoxon Signed Rank test [12] was used to
compare the average of the sleep measurements by the
consumer devices and that by the clinical device. Pearson
correlation coefficients were used to evaluate the linear
relationship between the consumer devices and the clinical
device.
IV. RESULTS
A. Data Preprocessing
In total 43 nights of sleep were collected from participant 1
and 49 nights of sleep were collected from participant 2. Many
data were missing due to device failures, especially for
Neuroon and SLEEP SCOPE. Some of the reasons for device
failures include: sensor misplacement (e.g. device sliding off
the face, poor electrode contact with the skin), battery issues
(e.g. device running out of battery during the measurement),
and user’s incompliance (e.g. terminating the measurement
midnight). After removing missing data pair wisely, we
obtained two datasets of 11 (Neuroon vs SLEEP SCOPE) and
14 (Fitbit vs SLEEP SCOPE) entries respectively for
participant 1, and two datasets of 22 (Neuroon vs SLEEP
SCOPE) and 21 (Fitbit vs SLEEP SCOPE) entries respectively
for participant 2. The basic statistics of sleep quality for both
Fitbit vs SLEEP a
-3.29 2.73 3.31 3.29 -3.29
SCOPE (Participant 1)
Neuroon vs SLEEP
2.75 -2.93 -1.33 2.93 2.67
SCOPE (Participant 1)
Fitbit vs SLEEP
-1.49 -0.35 2.80 3.98 -2.05
SCOPE (Participant 2)
Neuroon vs SLEEP
3.81 -4.04 -3.78 4.11 3.94
SCOPE (Participant 2)
a.
Bold indicates a significant difference between devices (p < 0.05).
B. Agreement between Consumer and Clinical Devices
The Bland-Altman plots on aggregated sleep metrics are
shown in Figure 1 (Neuroon vs SLEEP SCOPE) and Figure 2
(Fitbit vs SLEEP SCOPE). Generally, Fitbit achieved better
agreement to SLEEP SCOPE in comparison to Neuroon. For
participant 1, however, Neuroon yielded better agreement to
SLEEP SCOPE in terms of SOL. For both participants, good
agreement between Neuroon and SLEEP SCOPE occurred
when the sleep was good, i.e., TST > 400 min, WASO < 18
min, SOL < 30 min and SE > 80% [13].
We delved deeper into the trend of device difference as a
function of device mean and found that inter-personal
differences mainly manifested in the dimension of SOL. In
what follows, the term “device difference” refers to the
difference between the measurement of a consumer device
(either Neuroon or Fitbit) and the measurement of the clinical
device SLEEP SCOPE, while the term “device mean” refers to
the mean of the measurements by the two devices. According
to the Bland-Altman plots in Figure 2, device difference
increases as device mean decreases for participant 2, but no
trend was observed for participant 1. As in the case of Fitbit
shown in Figure 3, device difference increases as device mean
increases for participant 1, but no trend was observed for
participant 2.
C. Differences Between Consumer and Clinical Devices
The results of Wilcoxon Signed Rank (WSR) test is shown
in Table III (for TST, WASO, SOL, NAWK, SE). Bold
numbers indicate significant differences between devices (p <
0.05). There are statistically significant differences between
consumer and clinical devices in many sleep dimensions, but
the results are not equivalent for both participants. Neuroon
worked better on participant 1, and no statistically significant
difference between Neuroon and SLEEP SCOPE was found in
the dimension of SOL. In comparison, Fitbit worked better on
participant 2, because no statistically significant difference
was found in the dimension of TST and WASO.

Authorized licensed use limited to: Mapua University. Downloaded on May 01,2021 at 12:32:40 UTC from IEEE Xplore. Restrictions apply.
 2017 Tenth International Conference on Mobile Computing and Ubiquitous Network (ICMU)

D. Relationship between Consumer and Clinical Devices
The Pearson correlation coefficients were computed to
assess the linear relationship between the measurements by
consumer devices and those by the clinical device. The results
on TST, WASO, SOL, NAWK and SE are summarized in
Table IV. For participant 1, no significant correlation was
found between Neuroon/Fitbit and SLEEP SCOPE. For
participant 2, strong correlation was found between Fitbit and
SLEEP SCOPE in the dimension of TST (r = 0.72, p < 0.000),
and moderate correlation was found in terms of WASO (r =
0.43, p = 0.049). Moderate correlation was also found between
Neuroon and SLEEP SCOPE in multiple sleep dimensions for
participant 2, including WASO (r = 0.46, p = 0.029), SE (r =
0.42, p = 0.054), and Wake Ratio (r = 0.45, p = 0.036).
E. Discussions
The results presented above demonstrated that the validity
of Fitbit and Neuroon was not consistent between the two
participants.

(a) Bland-Altman plots between Neuroon and SLEEP SCOPE on aggregated sleep metrics for participant 1.

(b) Bland-Altman plots between Neuroon and SLEEP SCOPE on aggregated sleep metrics for participant 2.

Fig. 1. Bland-Altman plots of aggregated sleep metrics (TST, WASO, NAWK, SOL, SE) measured by Neuroon and SLEEP SCOPE for each participant. (The x-
axis is the mean of Neuroon and SLEEP SCOPE, and the y-axis is the difference between the two devices. Dotted line represents two standard deviations from the
mean.)

(a) Bland-Altman plots between Fitbit and SLEEP SCOPE on aggregated sleep metrics for participant 1.

(b) Bland-Altman plots between Fitbit and SLEEP SCOPE on aggregated sleep metrics for participant 2.

Fig. 2. Bland-Altman plots of aggregated sleep metrics (TST, WASO, NAWK, SOL, SE) measured by Fitbit and SLEEP SCOPE for each participant. (The x-
axis is the mean of Fitbit and SLEEP SCOPE, and the y-axis is the difference between the two devices. Dotted line represents two standard deviations from the
mean.)

Authorized licensed use limited to: Mapua University. Downloaded on May 01,2021 at 12:32:40 UTC from IEEE Xplore. Restrictions apply.
 2017 Tenth International Conference on Mobile Computing and Ubiquitous Network (ICMU)
TABLE IV. CORRLEATION ANALYSIS ON AGGREGATED SLEEP METRICS
TST WASO SOL
Fitbit vs SLEEP
0.25 0.25 0.04
SCOPE (Participant 1)
Neuroon vs SLEEP
0.42 0.09 -0.53
SCOPE (Participant 1)
a**
Fitbit vs SLEEP 0.72
0.43* 0.22
SCOPE (Participant 2) **
Neuroon vs SLEEP
0.36 0.46* -0.28
SCOPE (Participant 2)
a.
Bold indicates a significant correlation (*p” 0.05, **p” 0.01, ***p” 0.001, ****p<0.000).
Fitbit measurements completely deviated from those by
SLEEP SCOPE for participant 1, whereas for participant 2
Fitbit achieved good agreement and significant correlation to
SLEEP SCOPE in the dimensions of TST and WASO. An
investigation into the scatterplot between Fitbit and SLEEP
SCOPE on SOL showed that Fitbit underestimated SOL
compared to SLEEP SCOPE. On the other hand, Neuroon
showed good agreement to SLEEP SCOPE in the dimension
of SOL for participant 1, but agreed poorly for participant 2.
Noting that participant 1 had longer SOL than participant 2, it
is likely that participant 1 were simply lying still during the
process of falling asleep. Since Fitbit measures sleep based on
movement, it may estimate this segment as the time asleep
rather than the time to fall asleep. As contrast, Neuroon
measures sleep based on brainwave, it was able to accurate
classify this segment as the time to fall asleep even if the
participant was not moving. As for participant 2, the disparity
between Neuroon and SLEEP SCOPE in the dimension of
SOL may due to the fact that he often fell asleep while reading
in bed without putting on the Neuroon eye mask properly.
After a little while, he would wake up to put on the eye mask.
This may explain the overestimation of SOL for him.
In addition to the interpersonal differences, we also found
a common trend in the validity of Neuroon for both
participants. Good agreement between Neuroon and SLEEP
SCOPE occurred when the sleep quality of the user matched
the clinical definition of good sleep, which suggests that
wearable EEG sleep trackers may be more accurate when the
user has a good night of sleep.
Based on this analysis, we may conclude that the validity
of consumer sleep tracking devices is affected by users’ sleep
quality per se as well as users’ sleep habit. Fitbit may produce
less accurate sleep measurements for people who take long
time to fall asleep, while Neuroon may not work well for
people who read in bed. In the next step, we will recruit a
larger cohort to investigate the characteristics of people who
tend to generate inaccurate Fitbit and Neuroon measurements
as well as the role of geographical variables such as gender,
BMI and age. Based on such information, we may tune the
sleep analysis model for each type of user.
V. CONCLUSION
In this study we validated two newest consumer sleep tracking
devices, i.e., Fitbit Charge 2 and Neuroon wearable EEG eye
Authorized licensed use limited to: Mapua University. Downloaded on May 01,2021 at 12:32:40 UTC from IEEE Xplore. Restrictions apply.
mask, in comparison to a clinical device SLEEP SCOPE on
NAWK SE
two participants separately. The results demonstrated
significant interpersonal differences. On the one hand, the
0.08 0.23 Fitbit measurements completely deviated from the clinical
measurements for one participant, but agreed very well for
0.18 -0.16
another participant, especially in the dimension of TST. On the
other hand, the Neuroon measurements agreed well but were
0.02 0.23
uncorrelated to the clinical measurements for one participant
0.27 0.42*
(on SOL), and correlated significantly but agreed poorly for
another participant (on WASO, SE, Wake Ratio). Our study
revealed that the validity of consumer sleep tracking devices
varies from user to user. Such interpersonal differences should
be considered in future validation studies and in designing
better consumer sleeps tracking technologies.
REFERENCES
[1] Z. Liang, M. A. Chapa-Martell, B. Ploderer, “Inter-individual
differences in sleep quality: insights from mining wearable sleep-
tracking data,” IPSJ Technical Report, 2017-MBL-82(54), pp.1-6, 2017.
[2] H. P. A. van Dongen, K. M. Vitellaro, D. F. Dinges, “Individual
differences in adult human sleep and wakefulness: Leitmotif for a
research agenda,” Sleep, vol.28, no.4, p.479-496, 2005.
[3] J. Mantua, N. Gravel, and R. Spencer, “Reliability of sleep measures
from four personal health monitoring device compared to research-based
actigraphy and polysomnography,” Sensors, vol.16, 646, 2016.
[4] E. O. Lillie, B.Patay, J. Diamant, et al, “The n-of-1 clinical trial: the
ultimate strategy for individualizing medicine?” Personalized Medicine,
vol.8, no.2, p.161-173, 2011.
[5] J. C. Cappelleri, N. Ting, “A modified large-sample approach to
approximate interval estimation for a particular intraclass correlation
coefficient,” Statics in Medicine, vol.22, no.11, p.1861-1877, 2003.
[6] C. A. Kushida,M. R. Littner, T. M. Morgenthaler, et al. “Practice
parameters or the indications for polysomnography and related
procedures: an updated for 2005,” Sleep, vol.18, no.4, p.499-519, 2005.
[7] M. Littner, C. A. Kushida, W. M. Anderson, et al., “Practice parameters
for the role of actigraphy in the study of sleep and circadian rhythms: an
update for 2002,” Sleep, vol.26, no.3, pp.337-341, 2003.
[8] B. P. Kolla, S. Mansukhani, and M. P. Mansukhani, “Consumer sleep
tracking devices: a review of mechanisms, validity and utility,” Expert
Review of Medical Devices, vol.13, no.5, p.497-506, 2016.
[9] M. de Zambotti, S. Claudatos, S. Inkelis, et al, “Evaluation of a
consumer fitness-tracking device to assess sleep in adults,” Chronobiol
Int. vol.32, no.7, pp.1024-1028, 2015.
[10] D. Shrivastava, S. Jung, M. Saadat, et al. “How to interpret the results of
a sleep study,” Journal of Community Hospital Internal Medicine
Perspectives, vol.4, 24983, 2014.
[11] J. M. Bland, D. G. Altman, “Statistical methods for assessing agreement
between two methods of clinical measurement,” Lancet, vol.1, p.307-
310, 1986.
[12] F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics
Bulletin, vol.1, no.6, 80-83, 1945.
[13] M. M. Ohayon, M. A. Carskadon, C. Guilleminault, M. V. Vitiello,
“Meta-analysis of quantitative sleep parameters from childhood to old
age in healthy individuals: developing normative sleep values across the
human lifespan,” SLEEP, vol.27, no.7, 2004.
[14] Z. Liang, B. Ploderer, “Sleep tracking in the real world: a qualitative
study into barriers for improving sleep,” in Proceedings of the 28th
Australian Conference on Computer-Human Interaction, pp.537-541,
2016.
[15] W. Liu, B. Ploderer, and T. Hoang, “In bed with technology: challenges
and opportunities for sleep tracking,” in Proceedings of the Australian
Computer-Human Interaction Conference, pp. 142-151, 2015.